[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:44 dist-test-slave-dm3f krb5kdc[15731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:44 dist-test-slave-dm3f krb5kdc[15731](info): set up 2 sockets
Jun 19 18:52:44 dist-test-slave-dm3f krb5kdc[15731](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 18:52:45.703459 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.007s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:52:45 dist-test-slave-dm3f krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:52:45.888872 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:37739
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:37739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:46.243233 15744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:46.243580 15744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:46.243777 15744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:46.251650 15744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:46.251789 15744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:46.251895 15744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:46.252013 15744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:46.252108 15744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:46.263368 15744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:37739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:37739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:46.264866 15744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:46.266508 15744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:46.281232 15752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:46.281209 15750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:46.281173 15749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:46.282027 15744 server_base.cc:1034] running on GCE node
I20260619 18:52:46.283227 15744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:46.284960 15744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:46.286271 15744 hybrid_clock.cc:648] HybridClock initialized: now 1781895166286226 us; error 83 us; skew 500 ppm
I20260619 18:52:46.286774 15744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:46 dist-test-slave-dm3f krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895166, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:46.290362 15744 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:52:46.292388 15744 webserver.cc:460] Webserver started at http://127.15.91.62:34567/ using document root <none> and password file <none>
I20260619 18:52:46.293246 15744 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:46.293371 15744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:46.293666 15744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:46.297277 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "b3a02b9f35f8476f81a374d830219f3f"
format_stamp: "Formatted at 2026-06-19 18:52:46 on dist-test-slave-dm3f"
server_key: "8fe09612a61cd43a4e353039cb35598c"
server_key_iv: "72a1b8e050a22318a4c6b04948867b8b"
server_key_version: "encryptionkey@0"
I20260619 18:52:46.298193 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "b3a02b9f35f8476f81a374d830219f3f"
format_stamp: "Formatted at 2026-06-19 18:52:46 on dist-test-slave-dm3f"
server_key: "8fe09612a61cd43a4e353039cb35598c"
server_key_iv: "72a1b8e050a22318a4c6b04948867b8b"
server_key_version: "encryptionkey@0"
I20260619 18:52:46.304608 15744 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:52:46.309070 15759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:46.310916 15744 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:52:46.311129 15744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "b3a02b9f35f8476f81a374d830219f3f"
format_stamp: "Formatted at 2026-06-19 18:52:46 on dist-test-slave-dm3f"
server_key: "8fe09612a61cd43a4e353039cb35598c"
server_key_iv: "72a1b8e050a22318a4c6b04948867b8b"
server_key_version: "encryptionkey@0"
I20260619 18:52:46.311340 15744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:46.326030 15744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:46.331466 15744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:46.331746 15744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:46.353410 15744 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:37739
I20260619 18:52:46.353442 15810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:37739 every 8 connection(s)
I20260619 18:52:46.355266 15744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:46.358567 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 15744
I20260619 18:52:46.358984 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:52:46.360788 15811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:46.361701 15724 external_mini_cluster.cc:1428] Setting key a5cabc388c36fe10641f1a13e11f73a6
I20260619 18:52:46.374548 15811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f: Bootstrap starting.
I20260619 18:52:46.378618 15811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:46.380452 15811 log.cc:824] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:46.384390 15811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f: No bootstrap required, opened a new log
I20260619 18:52:46.394140 15811 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a02b9f35f8476f81a374d830219f3f" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37739 } }
I20260619 18:52:46.394582 15811 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:46.394762 15811 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a02b9f35f8476f81a374d830219f3f, State: Initialized, Role: FOLLOWER
I20260619 18:52:46.395498 15811 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a02b9f35f8476f81a374d830219f3f" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37739 } }
I20260619 18:52:46.395789 15811 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:46.395961 15811 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:46.396155 15811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [term 0 FOLLOWER]: Advancing to term 1
Jun 19 18:52:46 dist-test-slave-dm3f krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:52:46.399138 15811 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a02b9f35f8476f81a374d830219f3f" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37739 } }
I20260619 18:52:46.399874 15811 leader_election.cc:302] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3a02b9f35f8476f81a374d830219f3f; no voters:
I20260619 18:52:46.400718 15811 leader_election.cc:288] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:46.400875 15816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:46.402438 15816 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [term 1 LEADER]: Becoming Leader. State: Replica: b3a02b9f35f8476f81a374d830219f3f, State: Running, Role: LEADER
I20260619 18:52:46.403209 15816 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a02b9f35f8476f81a374d830219f3f" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37739 } }
I20260619 18:52:46.403774 15811 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:46.410002 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:46.367079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.15.91.62:37739)
0619 18:52:46.368234 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:46.368254 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:46.369832 (+ 1578us) server_negotiation.cc:407] Connection header received
0619 18:52:46.380462 (+ 10630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:46.380502 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:46.381127 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:46.381682 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:46.385357 (+ 3675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:46.387722 (+ 2365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:46.389798 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:46.391216 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:46.391476 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:46.398289 (+ 6813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:46.398302 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:46.398326 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:46.398390 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:46.403372 (+ 4982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:46.404613 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:46.404625 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:46.404632 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:46.404874 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:46.405592 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:46.405596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:46.405599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:46.406352 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:46.406705 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:46.407850 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0619 18:52:46.408337 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":514,"threads_started":1}
I20260619 18:52:46.413305 15818 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3a02b9f35f8476f81a374d830219f3f. Latest consensus state: current_term: 1 leader_uuid: "b3a02b9f35f8476f81a374d830219f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a02b9f35f8476f81a374d830219f3f" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37739 } } }
I20260619 18:52:46.413264 15817 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3a02b9f35f8476f81a374d830219f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a02b9f35f8476f81a374d830219f3f" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37739 } } }
I20260619 18:52:46.413851 15817 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:46.415917 15818 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:46.416404 15823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:46.422675 15823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:46.435688 15823 catalog_manager.cc:1347] Generated new cluster ID: 2f9f24863aea4ac2b0b436a8a3fcf92c
I20260619 18:52:46.435868 15823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:46.446012 15823 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:46.447839 15823 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:46.457779 15823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3a02b9f35f8476f81a374d830219f3f: Generated new TSK 0
I20260619 18:52:46.459144 15823 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:52:46.510505 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37739
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:46.880695 15837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:46.881045 15837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:46.881224 15837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:46.889545 15837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:46.889681 15837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:46.889851 15837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:52:46.901053 15837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:46.902566 15837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:46.904253 15837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:46.920853 15842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:46.920892 15843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:46.920853 15845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:46.923158 15837 server_base.cc:1034] running on GCE node
I20260619 18:52:46.924377 15837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:46.925900 15837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:46.927214 15837 hybrid_clock.cc:648] HybridClock initialized: now 1781895166927148 us; error 62 us; skew 500 ppm
I20260619 18:52:46.927673 15837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:46 dist-test-slave-dm3f krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895166, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:46.932312 15837 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:52:46.934952 15837 webserver.cc:460] Webserver started at http://127.15.91.1:35809/ using document root <none> and password file <none>
I20260619 18:52:46.935839 15837 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:46.935982 15837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:46.936287 15837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:46.939848 15837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "4db5985328a94c3fad6081816eac008a"
format_stamp: "Formatted at 2026-06-19 18:52:46 on dist-test-slave-dm3f"
server_key: "0bfeebfdd126de74e60eeae876a99373"
server_key_iv: "d02cc0db6163e22e18c6ea080535527b"
server_key_version: "encryptionkey@0"
I20260619 18:52:46.940815 15837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "4db5985328a94c3fad6081816eac008a"
format_stamp: "Formatted at 2026-06-19 18:52:46 on dist-test-slave-dm3f"
server_key: "0bfeebfdd126de74e60eeae876a99373"
server_key_iv: "d02cc0db6163e22e18c6ea080535527b"
server_key_version: "encryptionkey@0"
I20260619 18:52:46.947158 15837 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:52:46.951745 15852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:46.953558 15837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:52:46.953910 15837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "4db5985328a94c3fad6081816eac008a"
format_stamp: "Formatted at 2026-06-19 18:52:46 on dist-test-slave-dm3f"
server_key: "0bfeebfdd126de74e60eeae876a99373"
server_key_iv: "d02cc0db6163e22e18c6ea080535527b"
server_key_version: "encryptionkey@0"
I20260619 18:52:46.954175 15837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:46.979728 15837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:46.985152 15837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:46.985502 15837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:46.986663 15837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:52:46.988291 15837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:52:46.988423 15837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:46.988600 15837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:52:46.988677 15837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:47.027571 15837 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:42947
I20260619 18:52:47.027606 15964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:42947 every 8 connection(s)
I20260619 18:52:47.029035 15837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:52:47.032603 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 15837
I20260619 18:52:47.032903 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:52:47.033779 15724 external_mini_cluster.cc:1428] Setting key 21d4c1d7fb0cf45ecc24c0c25c83b959
Jun 19 18:52:47 dist-test-slave-dm3f krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895166, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:52:47.056993 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:47.032166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:42237 (local address 127.15.91.62:37739)
0619 18:52:47.032749 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:47.032756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:47.034095 (+ 1339us) server_negotiation.cc:407] Connection header received
0619 18:52:47.036364 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:47.036369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:47.036466 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:47.036579 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:47.038498 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:47.040077 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:47.042328 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:47.042911 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:47.043040 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:47.048751 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:47.048758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:47.048762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:47.048815 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:47.052491 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:47.053728 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:47.053735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:47.053739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:47.053923 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:47.054599 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:47.054604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:47.054606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:47.055088 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:47.055259 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:47.056460 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0619 18:52:47.056709 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":300,"threads_started":1}
I20260619 18:52:47.058831 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:47.032749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:37739 (local address 127.15.91.1:42237)
0619 18:52:47.033848 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:47.033914 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:47.035990 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:47.036918 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:47.036929 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:47.037556 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:47.038232 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:47.038253 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:47.040332 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:47.040337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:47.042047 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:47.042074 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:47.043198 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:47.043208 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:47.043622 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:47.044550 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:47.044588 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:47.048325 (+ 3737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:47.052754 (+ 4429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:47.052768 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:47.052796 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:47.053547 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:47.054143 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:47.054151 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:47.054154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:47.054403 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:47.055325 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:47.055336 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:47.055903 (+ 567us) client_negotiation.cc:769] Sending connection context
0619 18:52:47.056347 (+ 444us) client_negotiation.cc:240] Negotiation successful
0619 18:52:47.057101 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":402,"threads_started":1}
I20260619 18:52:47.060802 15965 heartbeater.cc:347] Connected to a master server at 127.15.91.62:37739
I20260619 18:52:47.061228 15965 heartbeater.cc:464] Registering TS with master...
I20260619 18:52:47.062115 15965 heartbeater.cc:511] Master 127.15.91.62:37739 requested a full tablet report, sending...
I20260619 18:52:47.064774 15776 ts_manager.cc:194] Registered new tserver with Master: 4db5985328a94c3fad6081816eac008a (127.15.91.1:42947)
I20260619 18:52:47.066370 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:42237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:52:47.078505 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37739
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:47.445796 15971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:47.446117 15971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:47.446225 15971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:47.454077 15971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:47.454224 15971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:47.454346 15971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:52:47.465234 15971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:47.466674 15971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:47.468164 15971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:47.483814 15976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:47.483744 15977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:47.484427 15979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:47.485865 15971 server_base.cc:1034] running on GCE node
I20260619 18:52:47.486865 15971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:47.488049 15971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:47.489333 15971 hybrid_clock.cc:648] HybridClock initialized: now 1781895167489282 us; error 58 us; skew 500 ppm
I20260619 18:52:47.489740 15971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:47 dist-test-slave-dm3f krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895167, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:47.493422 15971 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:52:47.495390 15971 webserver.cc:460] Webserver started at http://127.15.91.2:36517/ using document root <none> and password file <none>
I20260619 18:52:47.496255 15971 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:47.496456 15971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:47.496857 15971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:47.500360 15971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "dd6ba7aa34fb4a0cb92cfbf7bac24862"
format_stamp: "Formatted at 2026-06-19 18:52:47 on dist-test-slave-dm3f"
server_key: "4cceb3c56d3cc0ca081f22fb5a957b80"
server_key_iv: "83f85c41dbbbc8d21e9676cbbad05386"
server_key_version: "encryptionkey@0"
I20260619 18:52:47.501250 15971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "dd6ba7aa34fb4a0cb92cfbf7bac24862"
format_stamp: "Formatted at 2026-06-19 18:52:47 on dist-test-slave-dm3f"
server_key: "4cceb3c56d3cc0ca081f22fb5a957b80"
server_key_iv: "83f85c41dbbbc8d21e9676cbbad05386"
server_key_version: "encryptionkey@0"
I20260619 18:52:47.507946 15971 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:52:47.512334 15986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:47.514179 15971 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:52:47.514446 15971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "dd6ba7aa34fb4a0cb92cfbf7bac24862"
format_stamp: "Formatted at 2026-06-19 18:52:47 on dist-test-slave-dm3f"
server_key: "4cceb3c56d3cc0ca081f22fb5a957b80"
server_key_iv: "83f85c41dbbbc8d21e9676cbbad05386"
server_key_version: "encryptionkey@0"
I20260619 18:52:47.514693 15971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:47.532789 15971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:47.537597 15971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:47.537961 15971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:47.539194 15971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:52:47.540779 15971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:52:47.540872 15971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:47.541061 15971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:52:47.541116 15971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:47.581866 15971 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:40075
I20260619 18:52:47.581909 16098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:40075 every 8 connection(s)
I20260619 18:52:47.583472 15971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:52:47.589344 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 15971
I20260619 18:52:47.589570 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:52:47.590131 15724 external_mini_cluster.cc:1428] Setting key 66e499ef4716eae0223508d170bf51aa
Jun 19 18:52:47 dist-test-slave-dm3f krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895167, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
I20260619 18:52:47.614076 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:47.586543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:49909 (local address 127.15.91.62:37739)
0619 18:52:47.587114 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:47.587121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:47.588400 (+ 1279us) server_negotiation.cc:407] Connection header received
0619 18:52:47.591207 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:47.591214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:47.591318 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:47.591478 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:47.593565 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:47.594912 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:47.598867 (+ 3955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:47.599364 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:47.599519 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:47.605085 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:47.605096 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:47.605103 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:47.605168 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:47.609395 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:47.610512 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:47.610519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:47.610522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:47.610896 (+ 374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:47.611718 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:47.611726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:47.611729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:47.612282 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:47.612485 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:47.613377 (+ 892us) server_negotiation.cc:299] Negotiation successful
0619 18:52:47.613663 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":306,"threads_started":1}
I20260619 18:52:47.615468 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:47.587289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:37739 (local address 127.15.91.2:49909)
0619 18:52:47.588184 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:47.588236 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:47.590803 (+ 2567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:47.591797 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:47.591808 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:47.592594 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:47.593297 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:47.593321 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:47.595344 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:47.595352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:47.598458 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:47.598498 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:47.599711 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:47.599723 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:47.600183 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:47.601148 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:47.601189 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:47.604685 (+ 3496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:47.609657 (+ 4972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:47.609668 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:47.609686 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:47.610260 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:47.611142 (+ 882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:47.611146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:47.611149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:47.611454 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:47.612481 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:47.612492 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:47.612936 (+ 444us) client_negotiation.cc:769] Sending connection context
0619 18:52:47.613266 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 18:52:47.613923 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":341,"threads_started":1}
I20260619 18:52:47.617442 16099 heartbeater.cc:347] Connected to a master server at 127.15.91.62:37739
I20260619 18:52:47.617827 16099 heartbeater.cc:464] Registering TS with master...
I20260619 18:52:47.618772 16099 heartbeater.cc:511] Master 127.15.91.62:37739 requested a full tablet report, sending...
I20260619 18:52:47.620946 15776 ts_manager.cc:194] Registered new tserver with Master: dd6ba7aa34fb4a0cb92cfbf7bac24862 (127.15.91.2:40075)
I20260619 18:52:47.622372 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:49909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:52:47.634114 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37739
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:47.988319 16105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:47.988655 16105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:47.988760 16105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:47.996398 16105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:47.996544 16105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:47.996670 16105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:52:48.007036 16105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:48.008356 16105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:48.009732 16105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:48.024680 16113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:48.024852 16110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:48.024853 16111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:48.026022 16105 server_base.cc:1034] running on GCE node
I20260619 18:52:48.026983 16105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:48.028378 16105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:48.029675 16105 hybrid_clock.cc:648] HybridClock initialized: now 1781895168029596 us; error 76 us; skew 500 ppm
I20260619 18:52:48.030113 16105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:48 dist-test-slave-dm3f krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895168, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:48.034624 16105 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:52:48.036546 16105 webserver.cc:460] Webserver started at http://127.15.91.3:34393/ using document root <none> and password file <none>
I20260619 18:52:48.037384 16105 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:48.037554 16105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:48.037920 16105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:48.041267 16105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "367eb8402e3f416b9feae6e2893fd7e2"
format_stamp: "Formatted at 2026-06-19 18:52:48 on dist-test-slave-dm3f"
server_key: "b82ac59b984a5c303b7bf554ff84f072"
server_key_iv: "71c15f57b8753c80444eac7cee5f21c8"
server_key_version: "encryptionkey@0"
I20260619 18:52:48.042209 16105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "367eb8402e3f416b9feae6e2893fd7e2"
format_stamp: "Formatted at 2026-06-19 18:52:48 on dist-test-slave-dm3f"
server_key: "b82ac59b984a5c303b7bf554ff84f072"
server_key_iv: "71c15f57b8753c80444eac7cee5f21c8"
server_key_version: "encryptionkey@0"
I20260619 18:52:48.049157 16105 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:52:48.053675 16120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:48.055455 16105 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:52:48.055693 16105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "367eb8402e3f416b9feae6e2893fd7e2"
format_stamp: "Formatted at 2026-06-19 18:52:48 on dist-test-slave-dm3f"
server_key: "b82ac59b984a5c303b7bf554ff84f072"
server_key_iv: "71c15f57b8753c80444eac7cee5f21c8"
server_key_version: "encryptionkey@0"
I20260619 18:52:48.055950 16105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:48.070124 15965 heartbeater.cc:503] Master 127.15.91.62:37739 was elected leader, sending a full tablet report...
I20260619 18:52:48.083704 16105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:48.088587 16105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:48.088934 16105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:48.090044 16105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:52:48.091707 16105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:52:48.091826 16105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:48.091940 16105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:52:48.091979 16105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:48.129406 16105 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:35683
I20260619 18:52:48.129458 16232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:35683 every 8 connection(s)
I20260619 18:52:48.130999 16105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:52:48.132910 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 16105
I20260619 18:52:48.133118 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:52:48.133572 15724 external_mini_cluster.cc:1428] Setting key 9200efb1b260761a1151df7ed5aeda58
Jun 19 18:52:48 dist-test-slave-dm3f krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895168, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:52:48.155761 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:48.134032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:43305 (local address 127.15.91.62:37739)
0619 18:52:48.134612 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:48.134620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:48.138206 (+ 3586us) server_negotiation.cc:407] Connection header received
0619 18:52:48.138265 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:48.138268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:48.138340 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:48.138429 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:48.140392 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.141213 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.143053 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.143478 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.143592 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:48.147830 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:48.147836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:48.147839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:48.147884 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:48.151476 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.152892 (+ 1416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.152896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.152899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.153084 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.153771 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.153775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.153777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.154082 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:48.154224 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:48.155206 (+ 982us) server_negotiation.cc:299] Negotiation successful
0619 18:52:48.155431 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":274,"threads_started":1}
I20260619 18:52:48.156828 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:48.134771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:37739 (local address 127.15.91.3:43305)
0619 18:52:48.135976 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:48.136038 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:48.137884 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:48.138784 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:48.138796 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:48.139430 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:48.140133 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:48.140157 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.141395 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.141399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:48.142865 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:48.142883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.143716 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.143721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:48.143972 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:48.144799 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:48.144825 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:48.147447 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:48.151815 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:48.151829 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:48.151853 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:48.152589 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:48.153247 (+ 658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:48.153253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:48.153256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:48.153608 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:48.154255 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:48.154263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:48.154696 (+ 433us) client_negotiation.cc:769] Sending connection context
0619 18:52:48.155079 (+ 383us) client_negotiation.cc:240] Negotiation successful
0619 18:52:48.155568 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":606,"threads_started":1}
I20260619 18:52:48.158578 16233 heartbeater.cc:347] Connected to a master server at 127.15.91.62:37739
I20260619 18:52:48.159013 16233 heartbeater.cc:464] Registering TS with master...
I20260619 18:52:48.159873 16233 heartbeater.cc:511] Master 127.15.91.62:37739 requested a full tablet report, sending...
I20260619 18:52:48.161949 15776 ts_manager.cc:194] Registered new tserver with Master: 367eb8402e3f416b9feae6e2893fd7e2 (127.15.91.3:35683)
I20260619 18:52:48.163167 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:43305
I20260619 18:52:48.163421 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:52:48 dist-test-slave-dm3f krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895168, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:52:48 dist-test-slave-dm3f krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895168, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:52:48.211519 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:48.194616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.15.91.1:42947)
0619 18:52:48.195379 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:48.195387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:48.195440 (+ 53us) server_negotiation.cc:407] Connection header received
0619 18:52:48.195640 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:48.195648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:48.195899 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:48.196050 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:48.197605 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.198666 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.200157 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.200818 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.200944 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:48.205386 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:48.205442 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:48.205461 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:48.205513 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:48.208556 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.209391 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.209399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.209401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.209564 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.209995 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.209998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.210000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.210370 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:48.210637 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:48.210972 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 18:52:48.211197 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":415,"threads_started":1}
W20260619 18:52:48.212924 15880 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35070
Jun 19 18:52:48 dist-test-slave-dm3f krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:52:48 dist-test-slave-dm3f krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:52:48.261960 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:48.245430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.15.91.1:42947)
0619 18:52:48.245693 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:48.245702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:48.245868 (+ 166us) server_negotiation.cc:407] Connection header received
0619 18:52:48.246048 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:48.246052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:48.246162 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:48.246262 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:48.247844 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.248696 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.250103 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.250581 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.250703 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:48.255397 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:48.255403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:48.255407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:48.255454 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:48.258457 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.259533 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.259538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.259541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.259748 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.260296 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.260300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.260302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.260631 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:48.260861 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:48.261269 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 18:52:48.261582 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 18:52:48.263969 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 15837
I20260619 18:52:48.279084 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:48.265084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.15.91.1:42947)
0619 18:52:48.265372 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:48.265382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:48.265721 (+ 339us) server_negotiation.cc:407] Connection header received
0619 18:52:48.265980 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:48.265984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:48.266055 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:48.266172 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:48.267637 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.268364 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.269784 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.270221 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.270385 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:48.273112 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:48.273118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:48.273121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:48.273165 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:48.276016 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.276873 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.276879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.276883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.277053 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.277467 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.277473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.277476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.277764 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:48.278045 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:48.278406 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 18:52:48.278663 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 18:52:48.279870 15960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:48.370014 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 15837
I20260619 18:52:48.386863 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 15971
Jun 19 18:52:48 dist-test-slave-dm3f krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:52:48.404759 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:48.387876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.15.91.2:40075)
0619 18:52:48.388391 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:48.388398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:48.388424 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:52:48.388483 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:48.388490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:48.388721 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:48.388864 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:48.390153 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.391305 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.392665 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.393242 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.393361 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:48.398099 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:48.398118 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:48.398135 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:48.398180 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:48.401539 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.402417 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.402425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.402428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.402587 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.403120 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.403123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.403125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.403521 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:48.403783 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:48.404202 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 18:52:48.404426 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
I20260619 18:52:48.406183 16094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:48.491951 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 15971
I20260619 18:52:48.508698 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 16105
Jun 19 18:52:48 dist-test-slave-dm3f krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:52:48.527570 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:48.509881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.15.91.3:35683)
0619 18:52:48.510577 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:48.510584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:48.510617 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:52:48.510679 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:48.510685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:48.510989 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:48.511127 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:48.512456 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.513516 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.515136 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:48.515743 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:48.515934 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:48.520799 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:48.520816 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:48.520834 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:48.520880 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:48.524308 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.525166 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.525174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.525176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.525313 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:48.525696 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:48.525699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:48.525701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:48.526072 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:48.526315 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:48.526893 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 18:52:48.527243 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":370,"threads_started":1}
I20260619 18:52:48.529078 16228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:48.615893 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 16105
I20260619 18:52:48.632172 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 15744
I20260619 18:52:48.633399 15806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:48.739323 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 15744
2026-06-19T18:52:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4148 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:48 dist-test-slave-dm3f krb5kdc[16255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:48 dist-test-slave-dm3f krb5kdc[16255](info): set up 2 sockets
Jun 19 18:52:48 dist-test-slave-dm3f krb5kdc[16255](info): commencing operation
krb5kdc: starting...
W20260619 18:52:49.842981 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:52:49 dist-test-slave-dm3f krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:52:50.030995 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46001
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:50.384618 16268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:50.384950 16268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:50.385042 16268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:50.393230 16268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:50.393376 16268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:50.393424 16268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:50.393476 16268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:50.393522 16268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:50.404356 16268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:50.405786 16268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:50.407332 16268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:50.421144 16273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:50.421054 16276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:50.421118 16274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:50.422122 16268 server_base.cc:1034] running on GCE node
I20260619 18:52:50.423174 16268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:50.424997 16268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:50.426334 16268 hybrid_clock.cc:648] HybridClock initialized: now 1781895170426290 us; error 76 us; skew 500 ppm
I20260619 18:52:50.426909 16268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:50 dist-test-slave-dm3f krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895170, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:50.434867 16268 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:52:50.437031 16268 webserver.cc:460] Webserver started at http://127.15.91.62:43791/ using document root <none> and password file <none>
I20260619 18:52:50.437935 16268 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:50.438129 16268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:50.438501 16268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:50.442065 16268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "aa1a2f3faf7e474e8ca0529d0b2460dd"
format_stamp: "Formatted at 2026-06-19 18:52:50 on dist-test-slave-dm3f"
server_key: "db7a930e50c60b67baa16f29d43c3b2c"
server_key_iv: "87dd11e2a928ad2f28a486c8f858be4b"
server_key_version: "encryptionkey@0"
I20260619 18:52:50.443122 16268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "aa1a2f3faf7e474e8ca0529d0b2460dd"
format_stamp: "Formatted at 2026-06-19 18:52:50 on dist-test-slave-dm3f"
server_key: "db7a930e50c60b67baa16f29d43c3b2c"
server_key_iv: "87dd11e2a928ad2f28a486c8f858be4b"
server_key_version: "encryptionkey@0"
I20260619 18:52:50.449638 16268 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:52:50.454051 16283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:50.455818 16268 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:52:50.456090 16268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "aa1a2f3faf7e474e8ca0529d0b2460dd"
format_stamp: "Formatted at 2026-06-19 18:52:50 on dist-test-slave-dm3f"
server_key: "db7a930e50c60b67baa16f29d43c3b2c"
server_key_iv: "87dd11e2a928ad2f28a486c8f858be4b"
server_key_version: "encryptionkey@0"
I20260619 18:52:50.456364 16268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:50.470551 16268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:50.476034 16268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:50.476367 16268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:50.499810 16268 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:46001
I20260619 18:52:50.499814 16334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:46001 every 8 connection(s)
I20260619 18:52:50.501657 16268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:50.507150 16335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:50.511492 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 16268
I20260619 18:52:50.511785 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:52:50.512560 15724 external_mini_cluster.cc:1428] Setting key f150b9247aec214d908b4503fe161106
I20260619 18:52:50.520372 16335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd: Bootstrap starting.
I20260619 18:52:50.525252 16335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:50.526804 16335 log.cc:824] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:50.530319 16335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd: No bootstrap required, opened a new log
Jun 19 18:52:50 dist-test-slave-dm3f krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:52:50.539505 16335 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1a2f3faf7e474e8ca0529d0b2460dd" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46001 } }
I20260619 18:52:50.540082 16335 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:50.540268 16335 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1a2f3faf7e474e8ca0529d0b2460dd, State: Initialized, Role: FOLLOWER
I20260619 18:52:50.540930 16335 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1a2f3faf7e474e8ca0529d0b2460dd" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46001 } }
I20260619 18:52:50.541250 16335 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:50.541416 16335 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:50.541617 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:50.515322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.15.91.62:46001)
0619 18:52:50.516430 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:50.516447 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:50.516511 (+ 64us) server_negotiation.cc:407] Connection header received
0619 18:52:50.519043 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:50.519083 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:50.519653 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:50.520101 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:50.522480 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:50.524061 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:50.526178 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:50.527631 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:50.527933 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:50.532393 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:50.532403 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:50.532422 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:50.532481 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:50.536303 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:50.537257 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:50.537266 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:50.537274 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:50.537508 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:50.537958 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:50.537973 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:50.537976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:50.538647 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:50.539023 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:50.539602 (+ 579us) server_negotiation.cc:299] Negotiation successful
0619 18:52:50.540132 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":472,"threads_started":1}
I20260619 18:52:50.541625 16335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:50.545079 16335 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1a2f3faf7e474e8ca0529d0b2460dd" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46001 } }
I20260619 18:52:50.545841 16335 leader_election.cc:302] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa1a2f3faf7e474e8ca0529d0b2460dd; no voters:
I20260619 18:52:50.546423 16335 leader_election.cc:288] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:50.546581 16340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:50.547348 16340 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [term 1 LEADER]: Becoming Leader. State: Replica: aa1a2f3faf7e474e8ca0529d0b2460dd, State: Running, Role: LEADER
I20260619 18:52:50.547950 16340 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1a2f3faf7e474e8ca0529d0b2460dd" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46001 } }
I20260619 18:52:50.548867 16335 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:50.551196 16341 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa1a2f3faf7e474e8ca0529d0b2460dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1a2f3faf7e474e8ca0529d0b2460dd" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46001 } } }
I20260619 18:52:50.551604 16341 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:50.551893 16342 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa1a2f3faf7e474e8ca0529d0b2460dd. Latest consensus state: current_term: 1 leader_uuid: "aa1a2f3faf7e474e8ca0529d0b2460dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1a2f3faf7e474e8ca0529d0b2460dd" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46001 } } }
I20260619 18:52:50.552253 16342 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:50.554692 16347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:50.561805 16347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:50.572815 16347 catalog_manager.cc:1347] Generated new cluster ID: 55240fda00a7418fa22922b7b52e1b22
I20260619 18:52:50.572978 16347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:50.595912 16347 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:50.598075 16347 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:50.609633 16347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa1a2f3faf7e474e8ca0529d0b2460dd: Generated new TSK 0
I20260619 18:52:50.610913 16347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:52:50.657138 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46001
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:51.027122 16361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:51.027522 16361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:51.027714 16361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:51.035534 16361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:51.035723 16361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:51.035941 16361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:52:51.047425 16361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46001
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:51.048996 16361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:51.050875 16361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:51.066514 16369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:51.066571 16366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:51.066514 16367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:51.067710 16361 server_base.cc:1034] running on GCE node
I20260619 18:52:51.068487 16361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:51.069667 16361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:51.070935 16361 hybrid_clock.cc:648] HybridClock initialized: now 1781895171070881 us; error 55 us; skew 500 ppm
I20260619 18:52:51.071352 16361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:51 dist-test-slave-dm3f krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895171, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:51.075233 16361 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:52:51.077095 16361 webserver.cc:460] Webserver started at http://127.15.91.1:43433/ using document root <none> and password file <none>
I20260619 18:52:51.077888 16361 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:51.078012 16361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:51.078302 16361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:51.081619 16361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "4039795410dc4a0a9b6d9775a2a26d04"
format_stamp: "Formatted at 2026-06-19 18:52:51 on dist-test-slave-dm3f"
server_key: "c30889be422f9b361f4d506772d2865d"
server_key_iv: "2c8b5fd56bef2e8f3af8fe197dd2b812"
server_key_version: "encryptionkey@0"
I20260619 18:52:51.082615 16361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "4039795410dc4a0a9b6d9775a2a26d04"
format_stamp: "Formatted at 2026-06-19 18:52:51 on dist-test-slave-dm3f"
server_key: "c30889be422f9b361f4d506772d2865d"
server_key_iv: "2c8b5fd56bef2e8f3af8fe197dd2b812"
server_key_version: "encryptionkey@0"
I20260619 18:52:51.088927 16361 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 18:52:51.093536 16376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:51.095448 16361 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:52:51.095705 16361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "4039795410dc4a0a9b6d9775a2a26d04"
format_stamp: "Formatted at 2026-06-19 18:52:51 on dist-test-slave-dm3f"
server_key: "c30889be422f9b361f4d506772d2865d"
server_key_iv: "2c8b5fd56bef2e8f3af8fe197dd2b812"
server_key_version: "encryptionkey@0"
I20260619 18:52:51.095960 16361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:51.114537 16361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:51.119524 16361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:51.119858 16361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:51.120977 16361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:52:51.122538 16361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:52:51.122623 16361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:51.122828 16361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:52:51.122897 16361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:51.160727 16361 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:33029
I20260619 18:52:51.160737 16488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:33029 every 8 connection(s)
I20260619 18:52:51.162209 16361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:52:51.168859 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 16361
I20260619 18:52:51.169171 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:52:51.170003 15724 external_mini_cluster.cc:1428] Setting key e922a3946805b11c35677a4d58f8ac77
Jun 19 18:52:51 dist-test-slave-dm3f krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895171, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:52:51.188582 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:51.165131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:38677 (local address 127.15.91.62:46001)
0619 18:52:51.165631 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:51.165636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:51.166772 (+ 1136us) server_negotiation.cc:407] Connection header received
0619 18:52:51.168887 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:51.168893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:51.169005 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:51.169191 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:51.171637 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:51.172560 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:51.174391 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:51.174902 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:51.175061 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:51.180321 (+ 5260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:51.180328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:51.180331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:51.180375 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:51.183435 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:51.185048 (+ 1613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:51.185056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:51.185059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:51.185334 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:51.186236 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:51.186243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:51.186246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:51.186639 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:51.186933 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:51.187888 (+ 955us) server_negotiation.cc:299] Negotiation successful
0619 18:52:51.188170 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":390,"thread_start_us":284,"threads_started":1}
I20260619 18:52:51.189823 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:51.165639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46001 (local address 127.15.91.1:38677)
0619 18:52:51.166504 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:51.166570 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:51.168505 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:51.169605 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:51.169623 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:51.170567 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:51.171422 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:51.171454 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:51.172780 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:51.172785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:51.174180 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:51.174198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:51.175184 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:51.175194 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:51.175474 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:51.176309 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:51.176363 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:51.179922 (+ 3559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:51.183713 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:51.183729 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:51.183756 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:51.184677 (+ 921us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:51.185564 (+ 887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:51.185571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:51.185576 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:51.185958 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:51.186908 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:51.186919 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:51.187382 (+ 463us) client_negotiation.cc:769] Sending connection context
0619 18:52:51.187739 (+ 357us) client_negotiation.cc:240] Negotiation successful
0619 18:52:51.188380 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":353,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:52:51.192597 16489 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46001
I20260619 18:52:51.193224 16489 heartbeater.cc:464] Registering TS with master...
I20260619 18:52:51.194901 16489 heartbeater.cc:511] Master 127.15.91.62:46001 requested a full tablet report, sending...
I20260619 18:52:51.198042 16300 ts_manager.cc:194] Registered new tserver with Master: 4039795410dc4a0a9b6d9775a2a26d04 (127.15.91.1:33029)
I20260619 18:52:51.199854 16300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:38677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:52:51.218336 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46001
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:51.584714 16495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:51.585031 16495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:51.585134 16495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:51.592849 16495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:51.592984 16495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:51.593107 16495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:52:51.603786 16495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46001
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:51.605160 16495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:51.606600 16495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:51.621981 16500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:51.621886 16503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:51.622014 16501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:51.622993 16495 server_base.cc:1034] running on GCE node
I20260619 18:52:51.623731 16495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:51.624861 16495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:51.626154 16495 hybrid_clock.cc:648] HybridClock initialized: now 1781895171626076 us; error 75 us; skew 500 ppm
I20260619 18:52:51.626555 16495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:51 dist-test-slave-dm3f krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895171, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:51.630177 16495 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:52:51.632025 16495 webserver.cc:460] Webserver started at http://127.15.91.2:40471/ using document root <none> and password file <none>
I20260619 18:52:51.632911 16495 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:51.633095 16495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:51.633446 16495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:51.636868 16495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "d8df3783843347b382be920cc23285fa"
format_stamp: "Formatted at 2026-06-19 18:52:51 on dist-test-slave-dm3f"
server_key: "d3b025c8ddd3bf31f9658f6b844e3920"
server_key_iv: "acd188555c146399dd9641b35df26f52"
server_key_version: "encryptionkey@0"
I20260619 18:52:51.637856 16495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "d8df3783843347b382be920cc23285fa"
format_stamp: "Formatted at 2026-06-19 18:52:51 on dist-test-slave-dm3f"
server_key: "d3b025c8ddd3bf31f9658f6b844e3920"
server_key_iv: "acd188555c146399dd9641b35df26f52"
server_key_version: "encryptionkey@0"
I20260619 18:52:51.644065 16495 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:52:51.648309 16510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:51.650095 16495 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:52:51.650328 16495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "d8df3783843347b382be920cc23285fa"
format_stamp: "Formatted at 2026-06-19 18:52:51 on dist-test-slave-dm3f"
server_key: "d3b025c8ddd3bf31f9658f6b844e3920"
server_key_iv: "acd188555c146399dd9641b35df26f52"
server_key_version: "encryptionkey@0"
I20260619 18:52:51.650612 16495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:51.664875 16495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:51.670233 16495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:51.670616 16495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:51.671809 16495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:52:51.673475 16495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:52:51.673610 16495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:51.673777 16495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:52:51.673856 16495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:51.713027 16495 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:40077
I20260619 18:52:51.713076 16622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:40077 every 8 connection(s)
I20260619 18:52:51.714548 16495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:52:51.719713 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 16495
I20260619 18:52:51.720048 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:52:51.720592 15724 external_mini_cluster.cc:1428] Setting key f99a0fe2f7f9951bd34fa541ae64130a
Jun 19 18:52:51 dist-test-slave-dm3f krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895171, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
I20260619 18:52:51.742880 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:51.718415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:45183 (local address 127.15.91.62:46001)
0619 18:52:51.718993 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:51.718998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:51.720308 (+ 1310us) server_negotiation.cc:407] Connection header received
0619 18:52:51.722679 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:51.722683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:51.722832 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:51.722973 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:51.725551 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:51.726906 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:51.729656 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:51.730112 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:51.730275 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:51.735380 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:51.735387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:51.735390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:51.735433 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:51.738648 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:51.739732 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:51.739736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:51.739739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:51.739911 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:51.740550 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:51.740554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:51.740557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:51.740862 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:51.741072 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:51.742144 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0619 18:52:51.742492 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":316,"threads_started":1}
I20260619 18:52:51.744714 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:51.719115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46001 (local address 127.15.91.2:45183)
0619 18:52:51.720092 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:51.720155 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:51.722308 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:51.723458 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:51.723473 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:51.724330 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:51.725197 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:51.725225 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:51.727269 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:51.727279 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:51.729391 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:51.729424 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:51.730395 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:51.730414 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:51.730682 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:51.731670 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:51.731696 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:51.734965 (+ 3269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:51.738918 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:51.738931 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:51.738949 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:51.739570 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:51.740074 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:51.740082 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:51.740089 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:51.740382 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:51.741082 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:51.741094 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:51.741655 (+ 561us) client_negotiation.cc:769] Sending connection context
0619 18:52:51.742026 (+ 371us) client_negotiation.cc:240] Negotiation successful
0619 18:52:51.742930 (+ 904us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":430,"threads_started":1}
I20260619 18:52:51.746438 16623 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46001
I20260619 18:52:51.746928 16623 heartbeater.cc:464] Registering TS with master...
I20260619 18:52:51.747888 16623 heartbeater.cc:511] Master 127.15.91.62:46001 requested a full tablet report, sending...
I20260619 18:52:51.750053 16300 ts_manager.cc:194] Registered new tserver with Master: d8df3783843347b382be920cc23285fa (127.15.91.2:40077)
I20260619 18:52:51.751065 16300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:45183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:52:51.766435 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46001
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:52.121173 16629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:52.121521 16629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:52.121625 16629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:52.129292 16629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:52.129437 16629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:52.129565 16629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:52:52.139837 16629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46001
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:52.141283 16629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:52.142829 16629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:52.157852 16634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:52.157823 16635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:52.158353 16637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:52.159039 16629 server_base.cc:1034] running on GCE node
I20260619 18:52:52.159782 16629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:52.161271 16629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:52.162590 16629 hybrid_clock.cc:648] HybridClock initialized: now 1781895172162522 us; error 79 us; skew 500 ppm
I20260619 18:52:52.163086 16629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:52 dist-test-slave-dm3f krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895172, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:52.167922 16629 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:52:52.169894 16629 webserver.cc:460] Webserver started at http://127.15.91.3:41091/ using document root <none> and password file <none>
I20260619 18:52:52.170874 16629 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:52.171047 16629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:52.171422 16629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:52.174909 16629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "dbb2e29539f7495bad5c3019b6dc713c"
format_stamp: "Formatted at 2026-06-19 18:52:52 on dist-test-slave-dm3f"
server_key: "e95b501ec6da7a797b86cecefd632a73"
server_key_iv: "a5ffdc5b3805a3dbbeadbcd11f022f5e"
server_key_version: "encryptionkey@0"
I20260619 18:52:52.175928 16629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "dbb2e29539f7495bad5c3019b6dc713c"
format_stamp: "Formatted at 2026-06-19 18:52:52 on dist-test-slave-dm3f"
server_key: "e95b501ec6da7a797b86cecefd632a73"
server_key_iv: "a5ffdc5b3805a3dbbeadbcd11f022f5e"
server_key_version: "encryptionkey@0"
I20260619 18:52:52.182627 16629 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 18:52:52.187309 16644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:52.189098 16629 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:52:52.189344 16629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "dbb2e29539f7495bad5c3019b6dc713c"
format_stamp: "Formatted at 2026-06-19 18:52:52 on dist-test-slave-dm3f"
server_key: "e95b501ec6da7a797b86cecefd632a73"
server_key_iv: "a5ffdc5b3805a3dbbeadbcd11f022f5e"
server_key_version: "encryptionkey@0"
I20260619 18:52:52.189617 16629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:52.203522 16489 heartbeater.cc:503] Master 127.15.91.62:46001 was elected leader, sending a full tablet report...
I20260619 18:52:52.230012 16629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:52.235328 16629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:52.235635 16629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:52.236781 16629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:52:52.238379 16629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:52:52.238528 16629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:52.238688 16629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:52:52.238795 16629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 18:52:52.275914 16629 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:46369
I20260619 18:52:52.275983 16756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:46369 every 8 connection(s)
I20260619 18:52:52.277599 16629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:52:52.287194 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 16629
I20260619 18:52:52.287421 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:52:52.287978 15724 external_mini_cluster.cc:1428] Setting key c3717a34ecf0505351ace4e4d7490059
Jun 19 18:52:52 dist-test-slave-dm3f krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895172, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:52:52.303067 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.280823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:54373 (local address 127.15.91.62:46001)
0619 18:52:52.281348 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.281355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.282705 (+ 1350us) server_negotiation.cc:407] Connection header received
0619 18:52:52.284887 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.284893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.284994 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.285112 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:52.287053 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.288295 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.290795 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.291348 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.291455 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.295800 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:52.295807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:52.295811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:52.295855 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:52.299007 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.300034 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.300039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.300041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.300193 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.300915 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.300920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.300922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.301252 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:52.301396 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.302313 (+ 917us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.302610 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":234,"threads_started":1}
I20260619 18:52:52.304406 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.281582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46001 (local address 127.15.91.3:54373)
0619 18:52:52.282475 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:52.282535 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:52.284494 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:52.285387 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:52.285398 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:52.286050 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:52.286708 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.286799 (+ 91us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.288506 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.288510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.290525 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.290548 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.291624 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.291631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.291876 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.292632 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:52.292659 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:52.295495 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:52.299220 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:52.299231 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:52.299248 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:52.299852 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:52.300349 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:52.300354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:52.300356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:52.300721 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:52.301402 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:52.301409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:52.301820 (+ 411us) client_negotiation.cc:769] Sending connection context
0619 18:52:52.302104 (+ 284us) client_negotiation.cc:240] Negotiation successful
0619 18:52:52.302812 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":356,"threads_started":1}
I20260619 18:52:52.306057 16757 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46001
I20260619 18:52:52.306452 16757 heartbeater.cc:464] Registering TS with master...
I20260619 18:52:52.307395 16757 heartbeater.cc:511] Master 127.15.91.62:46001 requested a full tablet report, sending...
I20260619 18:52:52.309207 16300 ts_manager.cc:194] Registered new tserver with Master: dbb2e29539f7495bad5c3019b6dc713c (127.15.91.3:46369)
I20260619 18:52:52.310068 16300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:54373
I20260619 18:52:52.318131 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:52:52.339939 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.325333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.15.91.62:46001)
0619 18:52:52.325559 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.325565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.326173 (+ 608us) server_negotiation.cc:407] Connection header received
0619 18:52:52.326444 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.326449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.326521 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.326637 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:52.328322 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.329088 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.330523 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.331049 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.331211 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.333846 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:52.333853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:52.333856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:52.333899 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:52.336847 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.337696 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.337704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.337707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.337857 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.338273 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.338281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.338285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.338656 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:52.338996 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.339351 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.339610 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 18:52:52.347399 16300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:52:52.355183 16300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:52:52.397859 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.369730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34366 (local address 127.15.91.1:33029)
0619 18:52:52.388113 (+ 18383us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.388122 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.388147 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:52:52.388233 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.388242 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.388437 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.389499 (+ 1062us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:52.389794 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.391436 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.394313 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.395780 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.395920 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.396177 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.397281 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.397492 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18187,"thread_start_us":17998,"threads_started":1}
I20260619 18:52:52.397886 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.370311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:40077 (local address 127.0.0.1:50824)
0619 18:52:52.387641 (+ 17330us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:52.387667 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:52.387838 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:52.388630 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:52.388637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:52.388670 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:52.389264 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.389279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.391398 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.391404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.393383 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.393414 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.396315 (+ 2901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.396357 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.396885 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.396934 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 18:52:52.397081 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 18:52:52.397363 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17186,"mutex_wait_us":186,"thread_start_us":17011,"threads_started":1}
I20260619 18:52:52.397965 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.370790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58366 (local address 127.15.91.3:46369)
0619 18:52:52.379112 (+ 8322us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.379127 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.386827 (+ 7700us) server_negotiation.cc:407] Connection header received
0619 18:52:52.387306 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.387320 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.387700 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.387941 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:52.388792 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.390356 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.393791 (+ 3435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.395688 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.395855 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.396139 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.397367 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.397589 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8070,"thread_start_us":7879,"threads_started":1}
I20260619 18:52:52.398082 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.369992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:46369 (local address 127.0.0.1:58366)
0619 18:52:52.386639 (+ 16647us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:52.386662 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:52.387022 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:52.388142 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:52.388150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:52.388182 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:52.388511 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.388522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.390557 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.390564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.393582 (+ 3018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.393605 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.396381 (+ 2776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.396395 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.396805 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.396919 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 18:52:52.397082 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 18:52:52.397621 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16512,"mutex_wait_us":439,"thread_start_us":15499,"threads_started":1}
I20260619 18:52:52.398670 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.369525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:33029 (local address 127.0.0.1:34366)
0619 18:52:52.384917 (+ 15392us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:52.385005 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:52.385260 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:52.388658 (+ 3398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:52.388663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:52.388693 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:52.389241 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.389252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.391609 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.391615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.394047 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.394118 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.396454 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.396463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.396923 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.396958 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:52:52.397080 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 18:52:52.397365 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14946,"mutex_wait_us":258,"thread_start_us":16805,"threads_started":1}
I20260619 18:52:52.399171 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.370974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.15.91.2:40077)
0619 18:52:52.387090 (+ 16116us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.387104 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.387989 (+ 885us) server_negotiation.cc:407] Connection header received
0619 18:52:52.388105 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.388113 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.388362 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.388617 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:52.389479 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.391194 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.393735 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.395688 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.395855 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.396097 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.398566 (+ 2469us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.398832 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15921,"thread_start_us":15793,"threads_started":1}
I20260619 18:52:52.403038 16558 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef69eb39c884862a8c7a8f36fb29fb1 (DEFAULT_TABLE table=test-table [id=efda1a3b78ee42959f4cf7f1ff470b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:52:52.403036 16424 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef69eb39c884862a8c7a8f36fb29fb1 (DEFAULT_TABLE table=test-table [id=efda1a3b78ee42959f4cf7f1ff470b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:52:52.402906 16692 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef69eb39c884862a8c7a8f36fb29fb1 (DEFAULT_TABLE table=test-table [id=efda1a3b78ee42959f4cf7f1ff470b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:52:52.404515 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef69eb39c884862a8c7a8f36fb29fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:52.404620 16424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef69eb39c884862a8c7a8f36fb29fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:52.404620 16558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef69eb39c884862a8c7a8f36fb29fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:52.418185 16776 tablet_bootstrap.cc:492] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c: Bootstrap starting.
I20260619 18:52:52.421733 16778 tablet_bootstrap.cc:492] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa: Bootstrap starting.
I20260619 18:52:52.422823 16777 tablet_bootstrap.cc:492] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04: Bootstrap starting.
I20260619 18:52:52.425282 16776 tablet_bootstrap.cc:654] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:52.426913 16776 log.cc:824] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:52.427170 16777 tablet_bootstrap.cc:654] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:52.428588 16777 log.cc:824] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:52.430013 16778 tablet_bootstrap.cc:654] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:52.431212 16776 tablet_bootstrap.cc:492] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c: No bootstrap required, opened a new log
I20260619 18:52:52.431614 16776 ts_tablet_manager.cc:1397] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260619 18:52:52.431617 16778 log.cc:824] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:52.432137 16777 tablet_bootstrap.cc:492] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04: No bootstrap required, opened a new log
I20260619 18:52:52.432503 16777 ts_tablet_manager.cc:1397] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260619 18:52:52.435843 16778 tablet_bootstrap.cc:492] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa: No bootstrap required, opened a new log
I20260619 18:52:52.436215 16778 ts_tablet_manager.cc:1397] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.001s
I20260619 18:52:52.440939 16777 raft_consensus.cc:348] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4039795410dc4a0a9b6d9775a2a26d04" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 33029 } } peers { permanent_uuid: "dbb2e29539f7495bad5c3019b6dc713c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 46369 } } peers { permanent_uuid: "d8df3783843347b382be920cc23285fa" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40077 } }
I20260619 18:52:52.441453 16777 raft_consensus.cc:374] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:52.441615 16777 raft_consensus.cc:729] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4039795410dc4a0a9b6d9775a2a26d04, State: Initialized, Role: FOLLOWER
I20260619 18:52:52.442374 16777 consensus_queue.cc:260] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4039795410dc4a0a9b6d9775a2a26d04" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 33029 } } peers { permanent_uuid: "dbb2e29539f7495bad5c3019b6dc713c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 46369 } } peers { permanent_uuid: "d8df3783843347b382be920cc23285fa" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40077 } }
I20260619 18:52:52.443701 16776 raft_consensus.cc:348] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4039795410dc4a0a9b6d9775a2a26d04" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 33029 } } peers { permanent_uuid: "dbb2e29539f7495bad5c3019b6dc713c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 46369 } } peers { permanent_uuid: "d8df3783843347b382be920cc23285fa" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40077 } }
I20260619 18:52:52.444293 16776 raft_consensus.cc:374] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:52.444415 16776 raft_consensus.cc:729] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2e29539f7495bad5c3019b6dc713c, State: Initialized, Role: FOLLOWER
I20260619 18:52:52.444591 16777 ts_tablet_manager.cc:1428] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260619 18:52:52.445201 16776 consensus_queue.cc:260] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4039795410dc4a0a9b6d9775a2a26d04" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 33029 } } peers { permanent_uuid: "dbb2e29539f7495bad5c3019b6dc713c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 46369 } } peers { permanent_uuid: "d8df3783843347b382be920cc23285fa" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40077 } }
I20260619 18:52:52.446877 16757 heartbeater.cc:503] Master 127.15.91.62:46001 was elected leader, sending a full tablet report...
I20260619 18:52:52.447753 16776 ts_tablet_manager.cc:1428] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260619 18:52:52.450582 16778 raft_consensus.cc:348] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4039795410dc4a0a9b6d9775a2a26d04" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 33029 } } peers { permanent_uuid: "dbb2e29539f7495bad5c3019b6dc713c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 46369 } } peers { permanent_uuid: "d8df3783843347b382be920cc23285fa" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40077 } }
I20260619 18:52:52.451257 16778 raft_consensus.cc:374] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:52.451418 16778 raft_consensus.cc:729] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8df3783843347b382be920cc23285fa, State: Initialized, Role: FOLLOWER
I20260619 18:52:52.452266 16778 consensus_queue.cc:260] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4039795410dc4a0a9b6d9775a2a26d04" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 33029 } } peers { permanent_uuid: "dbb2e29539f7495bad5c3019b6dc713c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 46369 } } peers { permanent_uuid: "d8df3783843347b382be920cc23285fa" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40077 } }
I20260619 18:52:52.453656 16623 heartbeater.cc:503] Master 127.15.91.62:46001 was elected leader, sending a full tablet report...
I20260619 18:52:52.454398 16778 ts_tablet_manager.cc:1428] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa: Time spent starting tablet: real 0.018s user 0.009s sys 0.007s
W20260619 18:52:52.467453 16624 tablet.cc:2369] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:52:52.529635 16758 tablet.cc:2369] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:52:52.579800 16783 raft_consensus.cc:482] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:52:52.580111 16783 raft_consensus.cc:504] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4039795410dc4a0a9b6d9775a2a26d04" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 33029 } } peers { permanent_uuid: "dbb2e29539f7495bad5c3019b6dc713c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 46369 } } peers { permanent_uuid: "d8df3783843347b382be920cc23285fa" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40077 } }
I20260619 18:52:52.581945 16783 leader_election.cc:288] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4039795410dc4a0a9b6d9775a2a26d04 (127.15.91.1:33029), d8df3783843347b382be920cc23285fa (127.15.91.2:40077)
I20260619 18:52:52.589426 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.582540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:45645 (local address 127.15.91.2:40077)
0619 18:52:52.582828 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.582835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.582864 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:52:52.583086 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.583090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.583152 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.583224 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:52.583828 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.584989 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.587156 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.588448 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.588570 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.588652 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.589083 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.589194 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:52:52.589522 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.582412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:40077 (local address 127.15.91.3:45645)
0619 18:52:52.582666 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:52.582690 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:52.582887 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:52.583299 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:52.583308 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:52.583353 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:52.583639 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.583656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.585147 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.585153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.586930 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.586948 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.588653 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.588661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.588874 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.588923 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 18:52:52.589050 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 18:52:52.589205 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 18:52:52.590902 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.583059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:33029 (local address 127.15.91.3:59653)
0619 18:52:52.583799 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:52.583818 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:52.583966 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:52.584282 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:52.584288 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:52.584316 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:52.584679 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.584690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.586200 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.586204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.588230 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:52.588252 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.590109 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.590118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:52.590264 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.590290 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:52:52.590377 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 18:52:52.590487 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":3,"thread_start_us":417,"threads_started":1}
I20260619 18:52:52.590920 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.582512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:59653 (local address 127.15.91.1:33029)
0619 18:52:52.582714 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.582722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.583930 (+ 1208us) server_negotiation.cc:407] Connection header received
0619 18:52:52.584092 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.584096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.584159 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.584233 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:52.584894 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.586007 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.588493 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.589859 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.590013 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.590096 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.590453 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.590609 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:52:52.590765 16578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef69eb39c884862a8c7a8f36fb29fb1" candidate_uuid: "dbb2e29539f7495bad5c3019b6dc713c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8df3783843347b382be920cc23285fa" is_pre_election: true
I20260619 18:52:52.591432 16578 raft_consensus.cc:2454] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb2e29539f7495bad5c3019b6dc713c in term 0.
I20260619 18:52:52.592212 16645 leader_election.cc:302] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8df3783843347b382be920cc23285fa, dbb2e29539f7495bad5c3019b6dc713c; no voters:
I20260619 18:52:52.592136 16444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef69eb39c884862a8c7a8f36fb29fb1" candidate_uuid: "dbb2e29539f7495bad5c3019b6dc713c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4039795410dc4a0a9b6d9775a2a26d04" is_pre_election: true
I20260619 18:52:52.592671 16783 raft_consensus.cc:2781] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:52:52.592703 16444 raft_consensus.cc:2454] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb2e29539f7495bad5c3019b6dc713c in term 0.
I20260619 18:52:52.592852 16783 raft_consensus.cc:482] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:52:52.592960 16783 raft_consensus.cc:3037] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:52.595758 16783 raft_consensus.cc:504] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4039795410dc4a0a9b6d9775a2a26d04" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 33029 } } peers { permanent_uuid: "dbb2e29539f7495bad5c3019b6dc713c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 46369 } } peers { permanent_uuid: "d8df3783843347b382be920cc23285fa" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40077 } }
I20260619 18:52:52.596721 16783 leader_election.cc:288] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [CANDIDATE]: Term 1 election: Requested vote from peers 4039795410dc4a0a9b6d9775a2a26d04 (127.15.91.1:33029), d8df3783843347b382be920cc23285fa (127.15.91.2:40077)
I20260619 18:52:52.597287 16444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef69eb39c884862a8c7a8f36fb29fb1" candidate_uuid: "dbb2e29539f7495bad5c3019b6dc713c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4039795410dc4a0a9b6d9775a2a26d04"
I20260619 18:52:52.597378 16578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef69eb39c884862a8c7a8f36fb29fb1" candidate_uuid: "dbb2e29539f7495bad5c3019b6dc713c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8df3783843347b382be920cc23285fa"
I20260619 18:52:52.597563 16444 raft_consensus.cc:3037] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:52.597582 16578 raft_consensus.cc:3037] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:52.601125 16444 raft_consensus.cc:2454] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb2e29539f7495bad5c3019b6dc713c in term 1.
I20260619 18:52:52.601125 16578 raft_consensus.cc:2454] T 1ef69eb39c884862a8c7a8f36fb29fb1 P d8df3783843347b382be920cc23285fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb2e29539f7495bad5c3019b6dc713c in term 1.
I20260619 18:52:52.601848 16645 leader_election.cc:302] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8df3783843347b382be920cc23285fa, dbb2e29539f7495bad5c3019b6dc713c; no voters:
I20260619 18:52:52.602288 16783 raft_consensus.cc:2781] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:52.602903 16783 raft_consensus.cc:686] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [term 1 LEADER]: Becoming Leader. State: Replica: dbb2e29539f7495bad5c3019b6dc713c, State: Running, Role: LEADER
I20260619 18:52:52.603439 16783 consensus_queue.cc:237] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4039795410dc4a0a9b6d9775a2a26d04" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 33029 } } peers { permanent_uuid: "dbb2e29539f7495bad5c3019b6dc713c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 46369 } } peers { permanent_uuid: "d8df3783843347b382be920cc23285fa" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40077 } }
I20260619 18:52:52.608609 16299 catalog_manager.cc:5495] T 1ef69eb39c884862a8c7a8f36fb29fb1 P dbb2e29539f7495bad5c3019b6dc713c reported cstate change: term changed from 0 to 1, leader changed from <none> to dbb2e29539f7495bad5c3019b6dc713c (127.15.91.3). New cstate: current_term: 1 leader_uuid: "dbb2e29539f7495bad5c3019b6dc713c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4039795410dc4a0a9b6d9775a2a26d04" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 33029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbb2e29539f7495bad5c3019b6dc713c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 46369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8df3783843347b382be920cc23285fa" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40077 } health_report { overall_health: UNKNOWN } } }
Jun 19 18:52:52 dist-test-slave-dm3f krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:52:52.641081 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.624817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.15.91.1:33029)
0619 18:52:52.625083 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.625094 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.625512 (+ 418us) server_negotiation.cc:407] Connection header received
0619 18:52:52.625660 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.625665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.625735 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.625827 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:52.627556 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.628357 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.629706 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.630164 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.630276 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.634652 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:52.634671 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:52.634689 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:52.634809 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:52.637777 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.638634 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.638643 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.638645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.638906 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.639353 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.639357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.639359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.639761 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:52.640010 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.640449 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.640650 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 18:52:52.665339 16490 tablet.cc:2369] T 1ef69eb39c884862a8c7a8f36fb29fb1 P 4039795410dc4a0a9b6d9775a2a26d04: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 18:52:52 dist-test-slave-dm3f krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:52:52 dist-test-slave-dm3f krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:52:52.692237 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.676489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.15.91.1:33029)
0619 18:52:52.676774 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.676783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.677108 (+ 325us) server_negotiation.cc:407] Connection header received
0619 18:52:52.677269 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.677273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.677345 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.677451 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:52.679267 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.680019 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.681358 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.681823 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.681942 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.686446 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:52.686453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:52.686457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:52.686499 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:52.689267 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.690050 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.690054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.690056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.690202 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.690619 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.690622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.690625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.691001 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:52.691288 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.691608 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.691869 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 18:52:52 dist-test-slave-dm3f krb5kdc[16255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:52:52 dist-test-slave-dm3f krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:52:52.741246 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.726403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.15.91.1:33029)
0619 18:52:52.726702 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.726707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.726872 (+ 165us) server_negotiation.cc:407] Connection header received
0619 18:52:52.727037 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.727042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.727109 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.727200 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:52.728686 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.729408 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.730844 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.731304 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.731420 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.735768 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:52.735774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:52.735778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:52.735822 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:52.738435 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.739348 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.739353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.739355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.739519 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.739890 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.739893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.739895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.740201 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:52.740421 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.740663 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.740864 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260619 18:52:52.743255 16404 tablet_service.cc:2908] Rejecting scan request for tablet 1ef69eb39c884862a8c7a8f36fb29fb1: Uninitialized: clean time has not yet been initialized
I20260619 18:52:52.747287 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 16361
I20260619 18:52:52.761873 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.748409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.15.91.1:33029)
0619 18:52:52.748649 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.748657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.748982 (+ 325us) server_negotiation.cc:407] Connection header received
0619 18:52:52.749206 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.749212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.749293 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.749412 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:52.750827 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.751535 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.752834 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.753295 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.753466 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.756004 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:52.756010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:52.756013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:52.756066 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:52.758696 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.759586 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.759592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.759596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.759751 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.760160 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.760167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.760175 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.760519 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:52.760838 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.761234 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.761539 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:52:52.762595 16484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:52.855090 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 16361
I20260619 18:52:52.872665 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 16495
Jun 19 18:52:52 dist-test-slave-dm3f krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:52:52.889016 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.873573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.15.91.2:40077)
0619 18:52:52.873779 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:52.873788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:52.873811 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:52:52.873889 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:52.873893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:52.873969 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:52.874088 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:52.875475 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.876253 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.877697 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:52.878131 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:52.878286 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:52.882987 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:52.883004 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:52.883023 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:52.883073 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:52.886088 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.887035 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.887043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.887045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.887205 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:52.887602 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:52.887605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:52.887607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:52.888022 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:52.888289 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:52.888551 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 18:52:52.888741 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:52:52.889778 16618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:52.981392 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 16495
I20260619 18:52:52.998944 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 16629
Jun 19 18:52:53 dist-test-slave-dm3f krb5kdc[16255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:52:53.016294 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:52.999923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.15.91.3:46369)
0619 18:52:53.000482 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:53.000487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:53.000515 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:52:53.000575 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:53.000579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:53.000643 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:53.000749 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:53.002126 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:53.003016 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:53.004418 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:53.004847 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:53.004971 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:53.009679 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:53.009697 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:53.009714 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:53.009773 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:53.012926 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:53.013839 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:53.013846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:53.013849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:53.014025 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:53.014496 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:53.014501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:53.014503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:53.015024 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:53.015301 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:53.015622 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 18:52:53.015845 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":311,"threads_started":1}
I20260619 18:52:53.017081 16752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:53.109616 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 16629
I20260619 18:52:53.123769 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 16268
I20260619 18:52:53.125029 16330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:53.236932 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 16268
2026-06-19T18:52:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4498 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:53 dist-test-slave-dm3f krb5kdc[16807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:53 dist-test-slave-dm3f krb5kdc[16807](info): set up 2 sockets
Jun 19 18:52:53 dist-test-slave-dm3f krb5kdc[16807](info): commencing operation
krb5kdc: starting...
W20260619 18:52:54.339002 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:52:54 dist-test-slave-dm3f krb5kdc[16807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:52:54.535226 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:44983
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:44983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:54.886375 16820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:54.886797 16820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:54.886968 16820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:54.895282 16820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:54.895434 16820 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:52:54.895478 16820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:54.895596 16820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:54.895684 16820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:54.895802 16820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:54.907315 16820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:44983
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:44983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:54.908794 16820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:54.910300 16820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:54.924292 16826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:54.924352 16825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:54.924292 16828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:54.925314 16820 server_base.cc:1034] running on GCE node
I20260619 18:52:54.926159 16820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:54.927860 16820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:54.929126 16820 hybrid_clock.cc:648] HybridClock initialized: now 1781895174929065 us; error 35 us; skew 500 ppm
I20260619 18:52:54.929566 16820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:54 dist-test-slave-dm3f krb5kdc[16807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895174, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:54.933272 16820 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:52:54.935701 16820 webserver.cc:460] Webserver started at http://127.15.91.62:35809/ using document root <none> and password file <none>
I20260619 18:52:54.936756 16820 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:54.936930 16820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:54.937310 16820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:54.940857 16820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "d282e30c2c984eb79ab708d355ab8151"
format_stamp: "Formatted at 2026-06-19 18:52:54 on dist-test-slave-dm3f"
server_key: "24647bb921879e36ac7a3be219b5ba39"
server_key_iv: "c883ec0b6a87ae71074bcf89635a5480"
server_key_version: "encryptionkey@0"
I20260619 18:52:54.941829 16820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "d282e30c2c984eb79ab708d355ab8151"
format_stamp: "Formatted at 2026-06-19 18:52:54 on dist-test-slave-dm3f"
server_key: "24647bb921879e36ac7a3be219b5ba39"
server_key_iv: "c883ec0b6a87ae71074bcf89635a5480"
server_key_version: "encryptionkey@0"
I20260619 18:52:54.948563 16820 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:52:54.953477 16835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:54.955610 16820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:52:54.955917 16820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "d282e30c2c984eb79ab708d355ab8151"
format_stamp: "Formatted at 2026-06-19 18:52:54 on dist-test-slave-dm3f"
server_key: "24647bb921879e36ac7a3be219b5ba39"
server_key_iv: "c883ec0b6a87ae71074bcf89635a5480"
server_key_version: "encryptionkey@0"
I20260619 18:52:54.956223 16820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:54.989351 16820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:54.994261 16820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:54.994608 16820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:55.019814 16820 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:44983
I20260619 18:52:55.019830 16896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:44983 every 8 connection(s)
I20260619 18:52:55.021488 16820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:55.026060 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 16820
I20260619 18:52:55.026160 16897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:55.026324 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:52:55.027208 15724 external_mini_cluster.cc:1428] Setting key 0e4e51930badb41c865011c8339f9013
I20260619 18:52:55.042636 16897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151: Bootstrap starting.
Jun 19 18:52:55 dist-test-slave-dm3f krb5kdc[16807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:52:55.046921 16897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:55.048223 16897 log.cc:824] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:55.051911 16897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151: No bootstrap required, opened a new log
I20260619 18:52:55.056272 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:55.029550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.15.91.62:44983)
0619 18:52:55.030578 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:55.030592 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:55.030642 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:52:55.033254 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:55.033291 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:55.033846 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:55.034281 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:55.036320 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:55.037764 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:55.039876 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:55.040738 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:55.040935 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:55.046239 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:55.046250 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:55.046267 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:55.046321 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:55.050375 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:55.051444 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:55.051452 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:55.051460 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:55.051705 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:55.052249 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:55.052257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:55.052263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:55.053177 (+ 914us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:55.053567 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:55.054004 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 18:52:55.054588 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":488,"threads_started":1}
I20260619 18:52:55.061911 16897 raft_consensus.cc:348] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282e30c2c984eb79ab708d355ab8151" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44983 } }
I20260619 18:52:55.062345 16897 raft_consensus.cc:374] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:55.062489 16897 raft_consensus.cc:729] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d282e30c2c984eb79ab708d355ab8151, State: Initialized, Role: FOLLOWER
I20260619 18:52:55.063254 16897 consensus_queue.cc:260] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282e30c2c984eb79ab708d355ab8151" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44983 } }
I20260619 18:52:55.063558 16897 raft_consensus.cc:388] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:55.063730 16897 raft_consensus.cc:482] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:55.063947 16897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:55.066308 16897 raft_consensus.cc:504] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282e30c2c984eb79ab708d355ab8151" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44983 } }
I20260619 18:52:55.067094 16897 leader_election.cc:302] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d282e30c2c984eb79ab708d355ab8151; no voters:
I20260619 18:52:55.067792 16897 leader_election.cc:288] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:55.067956 16902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:55.068723 16902 raft_consensus.cc:686] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [term 1 LEADER]: Becoming Leader. State: Replica: d282e30c2c984eb79ab708d355ab8151, State: Running, Role: LEADER
I20260619 18:52:55.069343 16902 consensus_queue.cc:237] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282e30c2c984eb79ab708d355ab8151" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44983 } }
I20260619 18:52:55.070154 16897 sys_catalog.cc:564] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:55.072955 16904 sys_catalog.cc:455] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d282e30c2c984eb79ab708d355ab8151. Latest consensus state: current_term: 1 leader_uuid: "d282e30c2c984eb79ab708d355ab8151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282e30c2c984eb79ab708d355ab8151" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44983 } } }
I20260619 18:52:55.073570 16904 sys_catalog.cc:458] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:55.076272 16907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:55.076292 16903 sys_catalog.cc:455] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d282e30c2c984eb79ab708d355ab8151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282e30c2c984eb79ab708d355ab8151" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44983 } } }
I20260619 18:52:55.076785 16903 sys_catalog.cc:458] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:55.082839 16907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:55.094907 16907 catalog_manager.cc:1347] Generated new cluster ID: 6064ef1a679f49d2b07d26085d5a6a8f
I20260619 18:52:55.095060 16907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:55.112784 16907 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:55.114282 16907 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:55.127118 16907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151: Generated new TSK 0
I20260619 18:52:55.128175 16907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:52:55.187031 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44983
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:55.586889 16923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:55.587220 16923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:55.587401 16923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:55.595319 16923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:55.595467 16923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:55.595623 16923 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:52:55.595753 16923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:52:55.607277 16923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.91.62:44983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:55.608781 16923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:55.610301 16923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:55.626179 16928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:55.626399 16929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:55.626197 16931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:55.627372 16923 server_base.cc:1034] running on GCE node
I20260619 18:52:55.628170 16923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:55.629292 16923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:55.630595 16923 hybrid_clock.cc:648] HybridClock initialized: now 1781895175630506 us; error 76 us; skew 500 ppm
I20260619 18:52:55.630980 16923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:55 dist-test-slave-dm3f krb5kdc[16807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895175, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:55.639176 16923 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:52:55.641068 16923 webserver.cc:460] Webserver started at http://127.15.91.1:37093/ using document root <none> and password file <none>
I20260619 18:52:55.641916 16923 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:55.642079 16923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:55.642457 16923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:55.645955 16923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "113e72b8e9b84b6bb8104d1992a65384"
format_stamp: "Formatted at 2026-06-19 18:52:55 on dist-test-slave-dm3f"
server_key: "ab5a29320de2a06ab023c13a59e7556d"
server_key_iv: "6874a0bbd927867fa52b22ad208538f1"
server_key_version: "encryptionkey@0"
I20260619 18:52:55.647137 16923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "113e72b8e9b84b6bb8104d1992a65384"
format_stamp: "Formatted at 2026-06-19 18:52:55 on dist-test-slave-dm3f"
server_key: "ab5a29320de2a06ab023c13a59e7556d"
server_key_iv: "6874a0bbd927867fa52b22ad208538f1"
server_key_version: "encryptionkey@0"
I20260619 18:52:55.654142 16923 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 18:52:55.659416 16938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:55.661398 16923 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:52:55.661603 16923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "113e72b8e9b84b6bb8104d1992a65384"
format_stamp: "Formatted at 2026-06-19 18:52:55 on dist-test-slave-dm3f"
server_key: "ab5a29320de2a06ab023c13a59e7556d"
server_key_iv: "6874a0bbd927867fa52b22ad208538f1"
server_key_version: "encryptionkey@0"
I20260619 18:52:55.661834 16923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:55.682431 16923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:55.688172 16923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:55.688478 16923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:55.692992 16923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:52:55.693285 16923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:55.693523 16923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:52:55.693599 16923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:52:55 dist-test-slave-dm3f krb5kdc[16807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895175, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:52:55.722934 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:55.693541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:57091 (local address 127.15.91.62:44983)
0619 18:52:55.694257 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:55.694265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:55.695789 (+ 1524us) server_negotiation.cc:407] Connection header received
0619 18:52:55.698626 (+ 2837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:55.698633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:55.698798 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:55.698943 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:55.701935 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:55.703553 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:55.706841 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:55.707676 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:55.707884 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:55.715210 (+ 7326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:55.715221 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:55.715229 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:55.715286 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:55.719100 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:55.720114 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:55.720121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:55.720125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:55.720279 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:55.720840 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:55.720846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:55.720852 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:55.721170 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:55.721343 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:55.722212 (+ 869us) server_negotiation.cc:299] Negotiation successful
0619 18:52:55.722506 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":295,"threads_started":1}
I20260619 18:52:55.724638 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:55.694289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44983 (local address 127.15.91.1:57091)
0619 18:52:55.695527 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:55.695585 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:55.698162 (+ 2577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:55.699282 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:55.699292 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:55.700334 (+ 1042us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:55.701513 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:55.701546 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:55.703912 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:55.703922 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:55.706541 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:55.706576 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:55.708258 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:55.708272 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:55.709033 (+ 761us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:55.710156 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:55.710217 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:55.714789 (+ 4572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:55.719334 (+ 4545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:55.719352 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:55.719371 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:55.719938 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:55.720431 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:55.720436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:55.720440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:55.720700 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:55.721333 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:55.721342 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:55.721788 (+ 446us) client_negotiation.cc:769] Sending connection context
0619 18:52:55.722103 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 18:52:55.722768 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":522,"threads_started":1}
I20260619 18:52:55.754806 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:55.733951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44983 (local address 127.15.91.1:60477)
0619 18:52:55.734951 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:55.734986 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:55.735211 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:55.735751 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:55.735754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:55.736254 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:55.736629 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:55.736640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:55.738916 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:55.738924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:55.741431 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:55.741472 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:55.742891 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:55.742902 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:55.743162 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:55.743965 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:55.743985 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:55.744961 (+ 976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:55.749531 (+ 4570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:55.749539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:55.749547 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:55.750510 (+ 963us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:55.751198 (+ 688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:55.751209 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:55.751220 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:55.751472 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:55.752492 (+ 1020us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:55.752498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:55.752750 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 18:52:55.753516 (+ 766us) client_negotiation.cc:240] Negotiation successful
0619 18:52:55.754101 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":543,"threads_started":1}
I20260619 18:52:55.756273 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:55.733879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:60477 (local address 127.15.91.62:44983)
0619 18:52:55.734126 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:55.734135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:55.735142 (+ 1007us) server_negotiation.cc:407] Connection header received
0619 18:52:55.735390 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:55.735399 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:55.735525 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:55.735685 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:55.736836 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:55.738406 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:55.741699 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:55.742401 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:55.742556 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:55.745337 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:55.745345 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:55.745350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:55.745435 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:55.749248 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:55.750752 (+ 1504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:55.750760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:55.750766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:55.751014 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:55.751704 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:55.751713 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:55.751718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:55.752197 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:55.752416 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:55.755481 (+ 3065us) server_negotiation.cc:299] Negotiation successful
0619 18:52:55.755798 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 18:52:55.759218 16923 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:36551
I20260619 18:52:55.759289 17058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:36551 every 8 connection(s)
I20260619 18:52:55.761032 16923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:52:55.761945 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 16923
I20260619 18:52:55.762246 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:52:55.763326 15724 external_mini_cluster.cc:1428] Setting key 8170031827c88a409a09eb1073cd7f47
I20260619 18:52:55.766841 17059 heartbeater.cc:347] Connected to a master server at 127.15.91.62:44983
I20260619 18:52:55.767184 17059 heartbeater.cc:464] Registering TS with master...
I20260619 18:52:55.768186 17059 heartbeater.cc:511] Master 127.15.91.62:44983 requested a full tablet report, sending...
I20260619 18:52:55.771884 16852 ts_manager.cc:194] Registered new tserver with Master: 113e72b8e9b84b6bb8104d1992a65384 (127.15.91.1:36551)
I20260619 18:52:55.773972 16852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:57091
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:52:55.809988 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44983
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:56.167701 17063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:56.168017 17063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:56.168179 17063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:56.176148 17063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:56.176293 17063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:56.176473 17063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:52:56.176569 17063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:52:56.187769 17063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.91.62:44983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:56.189280 17063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:56.190836 17063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:56.206101 17071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:56.206056 17069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:56.206023 17068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:56.206885 17063 server_base.cc:1034] running on GCE node
I20260619 18:52:56.207557 17063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:56.208633 17063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:56.209896 17063 hybrid_clock.cc:648] HybridClock initialized: now 1781895176209818 us; error 46 us; skew 500 ppm
I20260619 18:52:56.210283 17063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:56 dist-test-slave-dm3f krb5kdc[16807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895176, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:56.214056 17063 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:52:56.215919 17063 webserver.cc:460] Webserver started at http://127.15.91.2:40403/ using document root <none> and password file <none>
I20260619 18:52:56.216753 17063 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:56.216863 17063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:56.217166 17063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:56.220762 17063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "56a477142e8548b59b6cfdadc914331f"
format_stamp: "Formatted at 2026-06-19 18:52:56 on dist-test-slave-dm3f"
server_key: "db664fe7453600cc617ec5a67a988398"
server_key_iv: "338b0948b6831d1c833142e7e8051f86"
server_key_version: "encryptionkey@0"
I20260619 18:52:56.221709 17063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "56a477142e8548b59b6cfdadc914331f"
format_stamp: "Formatted at 2026-06-19 18:52:56 on dist-test-slave-dm3f"
server_key: "db664fe7453600cc617ec5a67a988398"
server_key_iv: "338b0948b6831d1c833142e7e8051f86"
server_key_version: "encryptionkey@0"
I20260619 18:52:56.228410 17063 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 18:52:56.233212 17078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:56.235030 17063 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:52:56.235267 17063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "56a477142e8548b59b6cfdadc914331f"
format_stamp: "Formatted at 2026-06-19 18:52:56 on dist-test-slave-dm3f"
server_key: "db664fe7453600cc617ec5a67a988398"
server_key_iv: "338b0948b6831d1c833142e7e8051f86"
server_key_version: "encryptionkey@0"
I20260619 18:52:56.235553 17063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:56.271982 17063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:56.277032 17063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:56.277355 17063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:56.280686 17063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:52:56.280827 17063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:56.281037 17063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:52:56.281111 17063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:52:56 dist-test-slave-dm3f krb5kdc[16807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895176, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:52:56.314109 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:56.281208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:49373 (local address 127.15.91.62:44983)
0619 18:52:56.285141 (+ 3933us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:56.285146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:56.285170 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:52:56.287407 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:56.287412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:56.287501 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:56.287602 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:56.290643 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.291765 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.295238 (+ 3473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.295912 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.296079 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:56.303695 (+ 7616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:56.303706 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:56.303713 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:56.303784 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:56.308665 (+ 4881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:56.310435 (+ 1770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:56.310444 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:56.310449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:56.310685 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:56.311458 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:56.311467 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:56.311471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:56.311885 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:56.312094 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:56.313389 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0619 18:52:56.313673 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3807,"thread_start_us":3698,"threads_started":1}
I20260619 18:52:56.317121 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:56.282194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44983 (local address 127.15.91.2:49373)
0619 18:52:56.283611 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:56.283694 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:56.286919 (+ 3225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:56.287970 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:56.287985 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:56.288981 (+ 996us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:56.290269 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:56.290319 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.292099 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.292107 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:56.294892 (+ 2785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:56.294923 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.296282 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.296294 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:56.296824 (+ 530us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:56.298133 (+ 1309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:56.298182 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:56.303250 (+ 5068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:56.308974 (+ 5724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:56.308990 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:56.309022 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:56.310109 (+ 1087us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:56.310889 (+ 780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:56.310903 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:56.310909 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:56.311199 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:56.312088 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:56.312098 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:56.312737 (+ 639us) client_negotiation.cc:769] Sending connection context
0619 18:52:56.314510 (+ 1773us) client_negotiation.cc:240] Negotiation successful
0619 18:52:56.315269 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":563,"threads_started":1}
I20260619 18:52:56.338258 17063 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:46551
I20260619 18:52:56.338308 17197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:46551 every 8 connection(s)
I20260619 18:52:56.340155 17063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:52:56.341585 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 17063
I20260619 18:52:56.341806 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:52:56.342386 15724 external_mini_cluster.cc:1428] Setting key f14c65cd6f1c2ae64b54ef8c50b2a9b2
I20260619 18:52:56.343250 17199 heartbeater.cc:347] Connected to a master server at 127.15.91.62:44983
I20260619 18:52:56.343598 17199 heartbeater.cc:464] Registering TS with master...
I20260619 18:52:56.344659 17199 heartbeater.cc:511] Master 127.15.91.62:44983 requested a full tablet report, sending...
I20260619 18:52:56.347558 16852 ts_manager.cc:194] Registered new tserver with Master: 56a477142e8548b59b6cfdadc914331f (127.15.91.2:46551)
I20260619 18:52:56.348547 16852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:49373
I20260619 18:52:56.356745 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:56.338484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44983 (local address 127.15.91.2:50967)
0619 18:52:56.339290 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:56.339313 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:56.339478 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:56.339935 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:56.339939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:56.340559 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:56.340988 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:56.341001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.343125 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.343134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:56.345761 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:56.345790 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.347558 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.347564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:56.347815 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:56.348514 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:56.348540 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:56.349486 (+ 946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:56.353652 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:56.353659 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:56.353663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:56.354124 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:56.354640 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:56.354645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:56.354648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:56.354869 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:56.355731 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:56.355736 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:56.355936 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 18:52:56.356079 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 18:52:56.356404 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":475,"threads_started":1}
I20260619 18:52:56.357028 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:56.338327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:50967 (local address 127.15.91.62:44983)
0619 18:52:56.338611 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:56.338619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:56.339451 (+ 832us) server_negotiation.cc:407] Connection header received
0619 18:52:56.339647 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:56.339653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:56.339752 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:56.339881 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:56.341229 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.342838 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.346060 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.347080 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.347263 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:56.349761 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:56.349771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:56.349780 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:56.349832 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:56.353431 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:56.354285 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:56.354294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:56.354300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:56.354474 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:56.354994 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:56.355002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:56.355008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:56.355534 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:56.355712 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:56.356249 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 18:52:56.356568 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:52:56.388535 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44983
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:56.751155 17203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:56.751547 17203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:56.751734 17203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:56.760118 17203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:56.760291 17203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:56.760469 17203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:52:56.760618 17203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:52:56.771757 17203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.91.62:44983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:56.773191 17203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:56.774605 17203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:52:56.777527 17059 heartbeater.cc:503] Master 127.15.91.62:44983 was elected leader, sending a full tablet report...
W20260619 18:52:56.790314 17208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:56.790336 17211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:56.790304 17209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:56.791311 17203 server_base.cc:1034] running on GCE node
I20260619 18:52:56.792032 17203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:56.793107 17203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:56.794413 17203 hybrid_clock.cc:648] HybridClock initialized: now 1781895176794267 us; error 69 us; skew 500 ppm
I20260619 18:52:56.794960 17203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:56 dist-test-slave-dm3f krb5kdc[16807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895176, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:56.798671 17203 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:52:56.800612 17203 webserver.cc:460] Webserver started at http://127.15.91.3:33025/ using document root <none> and password file <none>
I20260619 18:52:56.801453 17203 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:56.801602 17203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:56.801973 17203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:56.805356 17203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "26ebdb0e48584abb835e39fd55d6a6c9"
format_stamp: "Formatted at 2026-06-19 18:52:56 on dist-test-slave-dm3f"
server_key: "abd597f9ada34745fc6fc02ba634633d"
server_key_iv: "c5dddbcc4244ac732635635c577d403b"
server_key_version: "encryptionkey@0"
I20260619 18:52:56.806474 17203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "26ebdb0e48584abb835e39fd55d6a6c9"
format_stamp: "Formatted at 2026-06-19 18:52:56 on dist-test-slave-dm3f"
server_key: "abd597f9ada34745fc6fc02ba634633d"
server_key_iv: "c5dddbcc4244ac732635635c577d403b"
server_key_version: "encryptionkey@0"
I20260619 18:52:56.812937 17203 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:52:56.817556 17218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:56.819571 17203 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:52:56.819823 17203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "26ebdb0e48584abb835e39fd55d6a6c9"
format_stamp: "Formatted at 2026-06-19 18:52:56 on dist-test-slave-dm3f"
server_key: "abd597f9ada34745fc6fc02ba634633d"
server_key_iv: "c5dddbcc4244ac732635635c577d403b"
server_key_version: "encryptionkey@0"
I20260619 18:52:56.820140 17203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:56.853259 17203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:56.858458 17203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:56.858803 17203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:56.862231 17203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:52:56.862368 17203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:56.862563 17203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:52:56.862660 17203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:52:56 dist-test-slave-dm3f krb5kdc[16807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895176, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:52:56.891245 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:56.863022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:60449 (local address 127.15.91.62:44983)
0619 18:52:56.863945 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:56.863950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:56.865013 (+ 1063us) server_negotiation.cc:407] Connection header received
0619 18:52:56.867694 (+ 2681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:56.867701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:56.867813 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:56.867946 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:56.870079 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.871125 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.874244 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.875165 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.875377 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:56.882822 (+ 7445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:56.882835 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:56.882843 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:56.882905 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:56.886543 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:56.887872 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:56.887879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:56.887883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:56.888057 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:56.888624 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:56.888630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:56.888633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:56.889028 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:56.889199 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:56.890484 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0619 18:52:56.890810 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":626,"threads_started":1}
I20260619 18:52:56.892458 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:56.863662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44983 (local address 127.15.91.3:60449)
0619 18:52:56.864754 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:56.864804 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:56.867297 (+ 2493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:56.868300 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:56.868319 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:56.869101 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:56.869790 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:56.869811 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.871369 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.871379 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:56.873910 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:56.873983 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.875882 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.875900 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:56.876497 (+ 597us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:56.877562 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:56.877615 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:56.882348 (+ 4733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:56.886807 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:56.886817 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:56.886839 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:56.887687 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:56.888217 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:56.888221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:56.888223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:56.888473 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:56.889199 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:56.889210 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:56.889829 (+ 619us) client_negotiation.cc:769] Sending connection context
0619 18:52:56.890346 (+ 517us) client_negotiation.cc:240] Negotiation successful
0619 18:52:56.891096 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":502,"threads_started":1}
I20260619 18:52:56.919837 17203 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:34813
I20260619 18:52:56.919999 17338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:34813 every 8 connection(s)
I20260619 18:52:56.921730 17203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:52:56.925576 17339 heartbeater.cc:347] Connected to a master server at 127.15.91.62:44983
I20260619 18:52:56.925921 17339 heartbeater.cc:464] Registering TS with master...
I20260619 18:52:56.926772 17339 heartbeater.cc:511] Master 127.15.91.62:44983 requested a full tablet report, sending...
I20260619 18:52:56.928830 16852 ts_manager.cc:194] Registered new tserver with Master: 26ebdb0e48584abb835e39fd55d6a6c9 (127.15.91.3:34813)
I20260619 18:52:56.929986 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:56.909243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:37499 (local address 127.15.91.62:44983)
0619 18:52:56.909450 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:56.909455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:56.910255 (+ 800us) server_negotiation.cc:407] Connection header received
0619 18:52:56.910513 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:56.910519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:56.910646 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:56.910774 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:56.913037 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.914371 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.917378 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.917991 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.918180 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:56.921583 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:56.921593 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:56.921597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:56.921665 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:56.926640 (+ 4975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:56.927557 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:56.927564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:56.927568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:56.927740 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:56.928274 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:56.928278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:56.928280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:56.928635 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:56.928774 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:56.929450 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 18:52:56.929699 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 18:52:56.930073 16852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:60449
I20260619 18:52:56.930226 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:56.909317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44983 (local address 127.15.91.3:37499)
0619 18:52:56.910106 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:56.910127 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:56.910355 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:56.910962 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:56.910967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:56.911916 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:56.912744 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:56.912769 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.914642 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.914653 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:56.917113 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:56.917144 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.918335 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.918346 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:56.918600 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:56.919862 (+ 1262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:56.919901 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:56.921194 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:56.926884 (+ 5690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:56.926891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:56.926896 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:56.927386 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:56.927879 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:56.927885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:56.927888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:56.928112 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:56.928774 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:56.928779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:56.929081 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 18:52:56.929253 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 18:52:56.929665 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":463,"threads_started":1}
I20260619 18:52:56.930855 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 17203
I20260619 18:52:56.931061 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:52:56.931618 15724 external_mini_cluster.cc:1428] Setting key 81ffbdd387896d6fd645ea018c1e4917
I20260619 18:52:56.935261 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:52:56 dist-test-slave-dm3f krb5kdc[16807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895176, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:52:56 dist-test-slave-dm3f krb5kdc[16807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895176, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:52:56.988752 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:56.971306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.15.91.62:44983)
0619 18:52:56.971582 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:56.971591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:56.971883 (+ 292us) server_negotiation.cc:407] Connection header received
0619 18:52:56.972036 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:56.972040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:56.972144 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:56.972245 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:56.973793 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.974576 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.976201 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:56.976694 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:56.976822 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:56.981888 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:56.981895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:56.981898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:56.981945 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:56.984857 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:56.985919 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:56.985926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:56.985928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:56.986174 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:56.986894 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:56.986899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:56.986901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:56.987312 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:56.987642 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:56.988094 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 18:52:56.988424 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:52:56.994575 16852 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:52:57.003407 16852 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:52:57.049396 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.019091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:46551 (local address 127.0.0.1:51090)
0619 18:52:57.037105 (+ 18014us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:57.037134 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:57.037257 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:57.039641 (+ 2384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:57.039649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:57.039681 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:57.040175 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.040190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.042551 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.042558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.045821 (+ 3263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.045847 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.048070 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.048078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.048380 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.048452 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 18:52:57.048619 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 18:52:57.048907 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17861,"mutex_wait_us":178,"thread_start_us":17718,"threads_started":1}
I20260619 18:52:57.049642 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.019033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.15.91.1:36551)
0619 18:52:57.037930 (+ 18897us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.037944 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.037993 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:52:57.038149 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.038162 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.038581 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.039815 (+ 1234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:57.040051 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.041589 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.045963 (+ 4374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.047969 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.048786 (+ 817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.049038 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.049116 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.049309 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18662,"thread_start_us":18506,"threads_started":1}
I20260619 18:52:57.050439 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.025079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.15.91.3:34813)
0619 18:52:57.038550 (+ 13471us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.038561 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.038604 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:52:57.038746 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.038755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.039020 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.040259 (+ 1239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:57.040623 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.042387 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.046953 (+ 4566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.048597 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.049525 (+ 928us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.049808 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.049892 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.050085 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13244,"thread_start_us":13032,"threads_started":1}
I20260619 18:52:57.050371 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.018801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:34813 (local address 127.0.0.1:57364)
0619 18:52:57.034290 (+ 15489us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:57.034382 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:57.034679 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:57.039286 (+ 4607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:57.039292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:57.039328 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:57.039956 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.039977 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.042551 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.042558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.045850 (+ 3292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.045860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.048879 (+ 3019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.048887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.049099 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.049139 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:52:57.049217 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:52:57.049389 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15043,"mutex_wait_us":292,"thread_start_us":17222,"threads_started":1}
I20260619 18:52:57.050508 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.018822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:36551 (local address 127.0.0.1:47110)
0619 18:52:57.036378 (+ 17556us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:57.036407 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:57.036543 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:57.038923 (+ 2380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:57.038932 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:57.038968 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:57.039506 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.039523 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.041775 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.041782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.045123 (+ 3341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.045146 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.048255 (+ 3109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.048264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.048495 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.048535 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:52:57.048613 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:52:57.050063 (+ 1450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17365,"mutex_wait_us":1304,"thread_start_us":16160,"threads_started":1}
I20260619 18:52:57.051733 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.019813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.15.91.2:46551)
0619 18:52:57.038850 (+ 19037us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.038860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.038890 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:52:57.039008 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.039015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.039236 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.040401 (+ 1165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:57.040609 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.042327 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.046099 (+ 3772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.047744 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.050858 (+ 3114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.051129 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.051212 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.051395 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18844,"thread_start_us":18686,"threads_started":1}
I20260619 18:52:57.055951 17129 tablet_service.cc:1467] Processing CreateTablet for tablet ed28a7edcde44b40b606e9c2efe6fda2 (DEFAULT_TABLE table=test-table [id=4ccf95173294415da48034283a8a0ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:52:57.055953 16989 tablet_service.cc:1467] Processing CreateTablet for tablet ed28a7edcde44b40b606e9c2efe6fda2 (DEFAULT_TABLE table=test-table [id=4ccf95173294415da48034283a8a0ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:52:57.057744 17129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed28a7edcde44b40b606e9c2efe6fda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:57.057734 16989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed28a7edcde44b40b606e9c2efe6fda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:57.058378 17269 tablet_service.cc:1467] Processing CreateTablet for tablet ed28a7edcde44b40b606e9c2efe6fda2 (DEFAULT_TABLE table=test-table [id=4ccf95173294415da48034283a8a0ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:52:57.060397 17269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed28a7edcde44b40b606e9c2efe6fda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:57.071827 17358 tablet_bootstrap.cc:492] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: Bootstrap starting.
I20260619 18:52:57.075671 17358 tablet_bootstrap.cc:654] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:57.077108 17358 log.cc:824] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:57.078481 17359 tablet_bootstrap.cc:492] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: Bootstrap starting.
I20260619 18:52:57.079146 17360 tablet_bootstrap.cc:492] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384: Bootstrap starting.
I20260619 18:52:57.082204 17358 tablet_bootstrap.cc:492] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: No bootstrap required, opened a new log
I20260619 18:52:57.082580 17358 ts_tablet_manager.cc:1397] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260619 18:52:57.083761 17359 tablet_bootstrap.cc:654] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:57.083761 17360 tablet_bootstrap.cc:654] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:57.085371 17360 log.cc:824] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:57.085371 17359 log.cc:824] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:57.091996 17359 tablet_bootstrap.cc:492] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: No bootstrap required, opened a new log
I20260619 18:52:57.092005 17360 tablet_bootstrap.cc:492] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384: No bootstrap required, opened a new log
I20260619 18:52:57.091949 17358 raft_consensus.cc:348] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } }
I20260619 18:52:57.092346 17360 ts_tablet_manager.cc:1397] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260619 18:52:57.092346 17359 ts_tablet_manager.cc:1397] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260619 18:52:57.092479 17358 raft_consensus.cc:374] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:57.092630 17358 raft_consensus.cc:729] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a477142e8548b59b6cfdadc914331f, State: Initialized, Role: FOLLOWER
I20260619 18:52:57.093315 17358 consensus_queue.cc:260] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } }
I20260619 18:52:57.095136 17364 raft_consensus.cc:482] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:52:57.095227 17199 heartbeater.cc:503] Master 127.15.91.62:44983 was elected leader, sending a full tablet report...
I20260619 18:52:57.095522 17358 ts_tablet_manager.cc:1428] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 18:52:57.096091 17364 raft_consensus.cc:504] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } }
W20260619 18:52:57.099366 17200 tablet.cc:2369] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:52:57.107285 17360 raft_consensus.cc:348] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } }
I20260619 18:52:57.107499 17359 raft_consensus.cc:348] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } }
I20260619 18:52:57.107882 17360 raft_consensus.cc:374] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:57.107996 17359 raft_consensus.cc:374] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:57.108014 17360 raft_consensus.cc:729] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113e72b8e9b84b6bb8104d1992a65384, State: Initialized, Role: FOLLOWER
I20260619 18:52:57.108099 17359 raft_consensus.cc:729] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ebdb0e48584abb835e39fd55d6a6c9, State: Initialized, Role: FOLLOWER
I20260619 18:52:57.108901 17359 consensus_queue.cc:260] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } }
I20260619 18:52:57.108902 17360 consensus_queue.cc:260] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } }
I20260619 18:52:57.116238 17364 leader_election.cc:288] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26ebdb0e48584abb835e39fd55d6a6c9 (127.15.91.3:34813), 113e72b8e9b84b6bb8104d1992a65384 (127.15.91.1:36551)
I20260619 18:52:57.119042 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.105066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:59755 (local address 127.15.91.3:34813)
0619 18:52:57.105277 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.105283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.107734 (+ 2451us) server_negotiation.cc:407] Connection header received
0619 18:52:57.108101 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.108108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.108201 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.108267 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:57.109195 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.110749 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.114938 (+ 4189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.116943 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.117121 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.117307 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.118326 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.118573 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:52:57.120311 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.104408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:34813 (local address 127.15.91.2:59755)
0619 18:52:57.107576 (+ 3168us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:57.107608 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:57.107888 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:57.108520 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:57.108525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:57.108553 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:57.108965 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.108981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.111068 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.111078 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.114533 (+ 3455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.114573 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.117683 (+ 3110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.117695 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.118037 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.118093 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 18:52:57.119474 (+ 1381us) client_negotiation.cc:240] Negotiation successful
0619 18:52:57.119736 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3013,"thread_start_us":2847,"threads_started":1}
I20260619 18:52:57.121809 17289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed28a7edcde44b40b606e9c2efe6fda2" candidate_uuid: "56a477142e8548b59b6cfdadc914331f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" is_pre_election: true
I20260619 18:52:57.122659 17289 raft_consensus.cc:2454] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56a477142e8548b59b6cfdadc914331f in term 0.
I20260619 18:52:57.123734 17079 leader_election.cc:302] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ebdb0e48584abb835e39fd55d6a6c9, 56a477142e8548b59b6cfdadc914331f; no voters:
I20260619 18:52:57.124420 17364 raft_consensus.cc:2781] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:52:57.124928 17364 raft_consensus.cc:482] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:52:57.125205 17364 raft_consensus.cc:3037] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:57.129058 17339 heartbeater.cc:503] Master 127.15.91.62:44983 was elected leader, sending a full tablet report...
I20260619 18:52:57.129495 17360 ts_tablet_manager.cc:1428] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384: Time spent starting tablet: real 0.037s user 0.018s sys 0.015s
I20260619 18:52:57.130000 17359 ts_tablet_manager.cc:1428] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: Time spent starting tablet: real 0.038s user 0.017s sys 0.019s
I20260619 18:52:57.130468 17364 raft_consensus.cc:504] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } }
I20260619 18:52:57.132071 17364 leader_election.cc:288] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [CANDIDATE]: Term 1 election: Requested vote from peers 26ebdb0e48584abb835e39fd55d6a6c9 (127.15.91.3:34813), 113e72b8e9b84b6bb8104d1992a65384 (127.15.91.1:36551)
I20260619 18:52:57.132544 17289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed28a7edcde44b40b606e9c2efe6fda2" candidate_uuid: "56a477142e8548b59b6cfdadc914331f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ebdb0e48584abb835e39fd55d6a6c9"
I20260619 18:52:57.132817 17289 raft_consensus.cc:3037] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:57.133888 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.112985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:34255 (local address 127.15.91.1:36551)
0619 18:52:57.113262 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.113273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.121566 (+ 8293us) server_negotiation.cc:407] Connection header received
0619 18:52:57.121923 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.121929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.122034 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.122110 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:57.122828 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.123881 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.130074 (+ 6193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.131390 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.131566 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.131673 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.133444 (+ 1771us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.133625 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:52:57.135108 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.113204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:36551 (local address 127.15.91.2:34255)
0619 18:52:57.121443 (+ 8239us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:57.121466 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:57.121729 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:57.122341 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:57.122349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:57.122384 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:57.122600 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.122612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.124084 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.124089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.126976 (+ 2887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.127014 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.131804 (+ 4790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.131813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.133132 (+ 1319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.133220 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 18:52:57.134551 (+ 1331us) client_negotiation.cc:240] Negotiation successful
0619 18:52:57.134787 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8097,"thread_start_us":17034,"threads_started":1}
I20260619 18:52:57.136682 17289 raft_consensus.cc:2454] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56a477142e8548b59b6cfdadc914331f in term 1.
I20260619 18:52:57.136790 17008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed28a7edcde44b40b606e9c2efe6fda2" candidate_uuid: "56a477142e8548b59b6cfdadc914331f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113e72b8e9b84b6bb8104d1992a65384"
I20260619 18:52:57.136705 17009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed28a7edcde44b40b606e9c2efe6fda2" candidate_uuid: "56a477142e8548b59b6cfdadc914331f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113e72b8e9b84b6bb8104d1992a65384" is_pre_election: true
I20260619 18:52:57.137243 17079 leader_election.cc:302] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ebdb0e48584abb835e39fd55d6a6c9, 56a477142e8548b59b6cfdadc914331f; no voters:
I20260619 18:52:57.137322 17008 raft_consensus.cc:3037] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:57.137720 17364 raft_consensus.cc:2781] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:57.137997 17364 raft_consensus.cc:686] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 1 LEADER]: Becoming Leader. State: Replica: 56a477142e8548b59b6cfdadc914331f, State: Running, Role: LEADER
I20260619 18:52:57.139925 17364 consensus_queue.cc:237] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } }
I20260619 18:52:57.141069 17008 raft_consensus.cc:2454] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56a477142e8548b59b6cfdadc914331f in term 1.
I20260619 18:52:57.144799 16852 catalog_manager.cc:5495] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f reported cstate change: term changed from 0 to 1, leader changed from <none> to 56a477142e8548b59b6cfdadc914331f (127.15.91.2). New cstate: current_term: 1 leader_uuid: "56a477142e8548b59b6cfdadc914331f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } health_report { overall_health: HEALTHY } } }
I20260619 18:52:57.174953 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.168153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.15.91.2:46551)
0619 18:52:57.168393 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.168398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.168427 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:52:57.168683 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.168687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.168783 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.168912 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:57.169326 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.170165 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.172098 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.172854 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.172992 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.173573 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:57.174185 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:57.174314 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.174476 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.174563 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 18:52:57.175333 17340 tablet.cc:2369] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:52:57.184880 17008 raft_consensus.cc:1261] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384 [term 1 FOLLOWER]: Refusing update from remote peer 56a477142e8548b59b6cfdadc914331f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:52:57.185032 17289 raft_consensus.cc:1261] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 FOLLOWER]: Refusing update from remote peer 56a477142e8548b59b6cfdadc914331f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:52:57.186062 17364 consensus_queue.cc:1035] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [LEADER]: Connected to new peer: Peer: permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:52:57.186445 17365 consensus_queue.cc:1035] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [LEADER]: Connected to new peer: Peer: permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:52:57.201871 17372 mvcc.cc:205] Tried to move back new op lower bound from 7298642645736632320 to 7298642645570224128. Current Snapshot: MvccSnapshot[applied={T|T < 7298642645736632320}]
I20260619 18:52:57.205467 17373 mvcc.cc:205] Tried to move back new op lower bound from 7298642645736632320 to 7298642645570224128. Current Snapshot: MvccSnapshot[applied={T|T < 7298642645736632320}]
I20260619 18:52:57.227324 16851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:52:57.227732 16851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:52:57.232650 16851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151: Sending DeleteTablet for 3 replicas of tablet ed28a7edcde44b40b606e9c2efe6fda2
I20260619 18:52:57.234465 17129 tablet_service.cc:1514] Processing DeleteTablet for tablet ed28a7edcde44b40b606e9c2efe6fda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:52:57 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:51090
I20260619 18:52:57.234450 17269 tablet_service.cc:1514] Processing DeleteTablet for tablet ed28a7edcde44b40b606e9c2efe6fda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:52:57 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:57364
I20260619 18:52:57.235423 16989 tablet_service.cc:1514] Processing DeleteTablet for tablet ed28a7edcde44b40b606e9c2efe6fda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:52:57 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:47110
I20260619 18:52:57.236109 17382 tablet_replica.cc:330] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: stopping tablet replica
I20260619 18:52:57.236706 17382 raft_consensus.cc:2229] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:52:57.237422 17382 raft_consensus.cc:2258] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:52:57.238494 16852 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:52:57.239310 16852 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:52:57.241264 17381 tablet_replica.cc:330] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: stopping tablet replica
I20260619 18:52:57.240481 17382 ts_tablet_manager.cc:1905] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:52:57.242177 17381 raft_consensus.cc:2229] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:52:57.243497 17383 tablet_replica.cc:330] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384: stopping tablet replica
I20260619 18:52:57.244319 17383 raft_consensus.cc:2229] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:52:57.244990 17383 raft_consensus.cc:2258] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:52:57.245959 17381 raft_consensus.cc:2258] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:52:57.248255 17383 ts_tablet_manager.cc:1905] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:52:57.252461 16988 tablet_service.cc:1467] Processing CreateTablet for tablet 85389e4af36d4d368a5314cd343e61a6 (DEFAULT_TABLE table=test-table [id=1dd89e5e65f444098780f7f4d6aa6a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:52:57.253055 16988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85389e4af36d4d368a5314cd343e61a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:57.253222 17382 ts_tablet_manager.cc:1918] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:52:57.253510 17382 log.cc:1196] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/ed28a7edcde44b40b606e9c2efe6fda2
I20260619 18:52:57.254200 17382 ts_tablet_manager.cc:1939] T ed28a7edcde44b40b606e9c2efe6fda2 P 26ebdb0e48584abb835e39fd55d6a6c9: Deleting consensus metadata
I20260619 18:52:57.254527 17269 tablet_service.cc:1467] Processing CreateTablet for tablet 85389e4af36d4d368a5314cd343e61a6 (DEFAULT_TABLE table=test-table [id=1dd89e5e65f444098780f7f4d6aa6a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:52:57.255092 17269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85389e4af36d4d368a5314cd343e61a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:57.257023 16836 catalog_manager.cc:4841] TS 26ebdb0e48584abb835e39fd55d6a6c9 (127.15.91.3:34813): tablet ed28a7edcde44b40b606e9c2efe6fda2 (table test-table [id=4ccf95173294415da48034283a8a0ae0]) successfully deleted
I20260619 18:52:57.259455 17129 tablet_service.cc:1467] Processing CreateTablet for tablet 85389e4af36d4d368a5314cd343e61a6 (DEFAULT_TABLE table=test-table [id=1dd89e5e65f444098780f7f4d6aa6a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:52:57.260268 17129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85389e4af36d4d368a5314cd343e61a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:57.260743 17360 tablet_bootstrap.cc:492] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384: Bootstrap starting.
I20260619 18:52:57.263108 17360 tablet_bootstrap.cc:654] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:57.266265 17359 tablet_bootstrap.cc:492] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9: Bootstrap starting.
I20260619 18:52:57.268190 17359 tablet_bootstrap.cc:654] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:57.269150 17381 ts_tablet_manager.cc:1905] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:52:57.273356 17358 tablet_bootstrap.cc:492] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f: Bootstrap starting.
I20260619 18:52:57.275331 17383 ts_tablet_manager.cc:1918] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:52:57.275586 17383 log.cc:1196] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/ed28a7edcde44b40b606e9c2efe6fda2
I20260619 18:52:57.276011 17358 tablet_bootstrap.cc:654] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:57.276242 17383 ts_tablet_manager.cc:1939] T ed28a7edcde44b40b606e9c2efe6fda2 P 113e72b8e9b84b6bb8104d1992a65384: Deleting consensus metadata
I20260619 18:52:57.278795 16839 catalog_manager.cc:4841] TS 113e72b8e9b84b6bb8104d1992a65384 (127.15.91.1:36551): tablet ed28a7edcde44b40b606e9c2efe6fda2 (table test-table [id=4ccf95173294415da48034283a8a0ae0]) successfully deleted
I20260619 18:52:57.284679 17381 ts_tablet_manager.cc:1918] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:52:57.285018 17381 log.cc:1196] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/ed28a7edcde44b40b606e9c2efe6fda2
I20260619 18:52:57.285024 17359 tablet_bootstrap.cc:492] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9: No bootstrap required, opened a new log
I20260619 18:52:57.285203 17359 ts_tablet_manager.cc:1397] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.002s
I20260619 18:52:57.285866 17381 ts_tablet_manager.cc:1939] T ed28a7edcde44b40b606e9c2efe6fda2 P 56a477142e8548b59b6cfdadc914331f: Deleting consensus metadata
I20260619 18:52:57.286533 17358 tablet_bootstrap.cc:492] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f: No bootstrap required, opened a new log
I20260619 18:52:57.286703 17358 ts_tablet_manager.cc:1397] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20260619 18:52:57.287070 17360 tablet_bootstrap.cc:492] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384: No bootstrap required, opened a new log
I20260619 18:52:57.287248 17360 ts_tablet_manager.cc:1397] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.002s
I20260619 18:52:57.287667 17358 raft_consensus.cc:348] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.287923 17358 raft_consensus.cc:374] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:57.288038 17358 raft_consensus.cc:729] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a477142e8548b59b6cfdadc914331f, State: Initialized, Role: FOLLOWER
I20260619 18:52:57.288463 16838 catalog_manager.cc:4841] TS 56a477142e8548b59b6cfdadc914331f (127.15.91.2:46551): tablet ed28a7edcde44b40b606e9c2efe6fda2 (table test-table [id=4ccf95173294415da48034283a8a0ae0]) successfully deleted
I20260619 18:52:57.288487 17358 consensus_queue.cc:260] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.289325 17359 raft_consensus.cc:348] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.289659 17359 raft_consensus.cc:374] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:57.289812 17359 raft_consensus.cc:729] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ebdb0e48584abb835e39fd55d6a6c9, State: Initialized, Role: FOLLOWER
I20260619 18:52:57.289816 17360 raft_consensus.cc:348] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.290084 17360 raft_consensus.cc:374] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:57.290199 17360 raft_consensus.cc:729] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113e72b8e9b84b6bb8104d1992a65384, State: Initialized, Role: FOLLOWER
I20260619 18:52:57.290258 17359 consensus_queue.cc:260] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.290477 17360 consensus_queue.cc:260] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.291190 17360 ts_tablet_manager.cc:1428] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:52:57.291198 17359 ts_tablet_manager.cc:1428] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260619 18:52:57.291970 17358 ts_tablet_manager.cc:1428] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260619 18:52:57.496567 17368 raft_consensus.cc:482] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:52:57.496862 17368 raft_consensus.cc:504] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.498852 17368 leader_election.cc:288] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 113e72b8e9b84b6bb8104d1992a65384 (127.15.91.1:36551), 56a477142e8548b59b6cfdadc914331f (127.15.91.2:46551)
I20260619 18:52:57.506702 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.499350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:37993 (local address 127.15.91.1:36551)
0619 18:52:57.499736 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.499744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.499764 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:52:57.500040 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.500047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.500135 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.500199 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:57.500880 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.501932 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.504275 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.505595 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.505749 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.505887 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.506230 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.506377 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260619 18:52:57.506804 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.499053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:36551 (local address 127.15.91.3:37993)
0619 18:52:57.499676 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:57.499696 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:57.499838 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:57.500421 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:57.500425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:57.500450 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:57.500655 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.500664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.502082 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.502086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.504067 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.504083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.505814 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.505821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.506032 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.506075 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 18:52:57.506201 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:52:57.506368 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":29,"thread_start_us":697,"threads_started":1}
I20260619 18:52:57.507661 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.499053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:46551 (local address 127.15.91.3:55019)
0619 18:52:57.500263 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:57.500281 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:57.500398 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:57.500773 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:57.500781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:57.500809 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:52:57.501136 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.501148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.502756 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.502762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.504615 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.504626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.506094 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.506102 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.506311 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.506349 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 18:52:57.507156 (+ 807us) client_negotiation.cc:240] Negotiation successful
0619 18:52:57.507318 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":578,"thread_start_us":322,"threads_started":1}
I20260619 18:52:57.507655 17008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85389e4af36d4d368a5314cd343e61a6" candidate_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113e72b8e9b84b6bb8104d1992a65384" is_pre_election: true
I20260619 18:52:57.507979 17008 raft_consensus.cc:2454] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ebdb0e48584abb835e39fd55d6a6c9 in term 0.
I20260619 18:52:57.508229 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.499307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:55019 (local address 127.15.91.2:46551)
0619 18:52:57.499582 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.499590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.500384 (+ 794us) server_negotiation.cc:407] Connection header received
0619 18:52:57.500521 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.500526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.500611 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.500726 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:52:57.501313 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.502555 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.504793 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.505670 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.505864 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.505983 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.506536 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.506770 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260619 18:52:57.508574 17222 leader_election.cc:302] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113e72b8e9b84b6bb8104d1992a65384, 26ebdb0e48584abb835e39fd55d6a6c9; no voters:
I20260619 18:52:57.509050 17368 raft_consensus.cc:2781] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:52:57.509182 17368 raft_consensus.cc:482] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:52:57.509276 17368 raft_consensus.cc:3037] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:57.509104 17149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85389e4af36d4d368a5314cd343e61a6" candidate_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a477142e8548b59b6cfdadc914331f" is_pre_election: true
I20260619 18:52:57.509665 17149 raft_consensus.cc:2454] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ebdb0e48584abb835e39fd55d6a6c9 in term 0.
I20260619 18:52:57.512805 17368 raft_consensus.cc:504] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.513731 17368 leader_election.cc:288] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [CANDIDATE]: Term 1 election: Requested vote from peers 113e72b8e9b84b6bb8104d1992a65384 (127.15.91.1:36551), 56a477142e8548b59b6cfdadc914331f (127.15.91.2:46551)
I20260619 18:52:57.514791 17149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85389e4af36d4d368a5314cd343e61a6" candidate_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a477142e8548b59b6cfdadc914331f"
I20260619 18:52:57.514783 17008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85389e4af36d4d368a5314cd343e61a6" candidate_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113e72b8e9b84b6bb8104d1992a65384"
I20260619 18:52:57.515095 17008 raft_consensus.cc:3037] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:57.515136 17149 raft_consensus.cc:3037] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:57.518201 17149 raft_consensus.cc:2454] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ebdb0e48584abb835e39fd55d6a6c9 in term 1.
I20260619 18:52:57.518612 17008 raft_consensus.cc:2454] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ebdb0e48584abb835e39fd55d6a6c9 in term 1.
I20260619 18:52:57.519335 17221 leader_election.cc:302] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 113e72b8e9b84b6bb8104d1992a65384, 26ebdb0e48584abb835e39fd55d6a6c9, 56a477142e8548b59b6cfdadc914331f; no voters:
W20260619 18:52:57.519709 17060 tablet.cc:2369] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:52:57.519984 17368 raft_consensus.cc:2781] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:57.520530 17368 raft_consensus.cc:686] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 LEADER]: Becoming Leader. State: Replica: 26ebdb0e48584abb835e39fd55d6a6c9, State: Running, Role: LEADER
I20260619 18:52:57.521009 17368 consensus_queue.cc:237] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.526254 16851 catalog_manager.cc:5495] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26ebdb0e48584abb835e39fd55d6a6c9 (127.15.91.3). New cstate: current_term: 1 leader_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } health_report { overall_health: HEALTHY } } }
Jun 19 18:52:57 dist-test-slave-dm3f krb5kdc[16807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895174, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:52:57.609917 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.589457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44983 (local address 127.0.0.1:40538)
0619 18:52:57.591458 (+ 2001us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:57.591486 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:57.591688 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:57.592230 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:57.592241 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:57.593260 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:57.593561 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.593577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.594971 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.594977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.596768 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.596793 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.597752 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.597759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.597945 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.598868 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:57.598897 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:57.602503 (+ 3606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:57.606464 (+ 3961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:57.606490 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:57.606502 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:57.607293 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:57.607805 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:57.607810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:57.607814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:57.608006 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:57.608636 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:57.608646 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:57.608866 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 18:52:57.609093 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 18:52:57.609444 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1767,"thread_start_us":1549,"threads_started":1}
I20260619 18:52:57.610193 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.589538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.15.91.62:44983)
0619 18:52:57.591859 (+ 2321us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.591864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.591889 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:52:57.591940 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.591943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.592010 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.592118 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:57.593826 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.594671 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.596990 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.597466 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.597603 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.603079 (+ 5476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:57.603085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:57.603089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:57.603127 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:57.606235 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:57.607473 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:57.607478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:57.607481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:57.607659 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:57.608134 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:57.608137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:57.608140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:57.608437 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:57.608609 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.609339 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.609765 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2199,"thread_start_us":2053,"threads_started":1}
I20260619 18:52:57.616681 16851 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:40538:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:52:57.618157 16851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:52:57.627156 16988 tablet_service.cc:1467] Processing CreateTablet for tablet 59bf26151ada4e39b716ab90de993230 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42ea1e09419d40be97c94c060a09628f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:52:57.627329 17129 tablet_service.cc:1467] Processing CreateTablet for tablet 59bf26151ada4e39b716ab90de993230 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42ea1e09419d40be97c94c060a09628f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:52:57.627816 16988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59bf26151ada4e39b716ab90de993230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:57.628129 17129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59bf26151ada4e39b716ab90de993230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:57.628932 17269 tablet_service.cc:1467] Processing CreateTablet for tablet 59bf26151ada4e39b716ab90de993230 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42ea1e09419d40be97c94c060a09628f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:52:57.629550 17269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59bf26151ada4e39b716ab90de993230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:57.636854 17360 tablet_bootstrap.cc:492] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384: Bootstrap starting.
I20260619 18:52:57.636855 17358 tablet_bootstrap.cc:492] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f: Bootstrap starting.
I20260619 18:52:57.638947 17360 tablet_bootstrap.cc:654] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:57.638959 17358 tablet_bootstrap.cc:654] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:57.640332 17359 tablet_bootstrap.cc:492] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: Bootstrap starting.
I20260619 18:52:57.642354 17359 tablet_bootstrap.cc:654] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:57.644933 17360 tablet_bootstrap.cc:492] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384: No bootstrap required, opened a new log
I20260619 18:52:57.645176 17360 ts_tablet_manager.cc:1397] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260619 18:52:57.645659 17358 tablet_bootstrap.cc:492] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f: No bootstrap required, opened a new log
I20260619 18:52:57.645861 17358 ts_tablet_manager.cc:1397] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260619 18:52:57.646039 17360 raft_consensus.cc:348] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.646296 17360 raft_consensus.cc:374] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:57.646377 17360 raft_consensus.cc:729] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113e72b8e9b84b6bb8104d1992a65384, State: Initialized, Role: FOLLOWER
I20260619 18:52:57.646644 17360 consensus_queue.cc:260] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.646782 17358 raft_consensus.cc:348] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.647097 17358 raft_consensus.cc:374] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:57.647198 17358 raft_consensus.cc:729] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a477142e8548b59b6cfdadc914331f, State: Initialized, Role: FOLLOWER
I20260619 18:52:57.647537 17367 tablet_replica.cc:439] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } }
I20260619 18:52:57.647922 17367 tablet_replica.cc:442] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:52:57.648479 17360 ts_tablet_manager.cc:1428] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 18:52:57.649911 17359 tablet_bootstrap.cc:492] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: No bootstrap required, opened a new log
I20260619 18:52:57.650228 17359 ts_tablet_manager.cc:1397] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260619 18:52:57.650345 17358 consensus_queue.cc:260] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.650981 17364 tablet_replica.cc:439] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } }
I20260619 18:52:57.651317 17364 tablet_replica.cc:442] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:52:57.651504 17359 raft_consensus.cc:348] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.651858 17359 raft_consensus.cc:374] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:57.651996 17359 raft_consensus.cc:729] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ebdb0e48584abb835e39fd55d6a6c9, State: Initialized, Role: FOLLOWER
I20260619 18:52:57.652330 17359 consensus_queue.cc:260] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.652884 17368 tablet_replica.cc:439] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } }
I20260619 18:52:57.653208 17368 tablet_replica.cc:442] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:52:57.653791 17359 ts_tablet_manager.cc:1428] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:52:57.654659 17358 ts_tablet_manager.cc:1428] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20260619 18:52:57.718497 17368 raft_consensus.cc:482] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:52:57.718781 17368 raft_consensus.cc:504] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.719558 17368 leader_election.cc:288] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 113e72b8e9b84b6bb8104d1992a65384 (127.15.91.1:36551), 56a477142e8548b59b6cfdadc914331f (127.15.91.2:46551)
I20260619 18:52:57.720281 17008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59bf26151ada4e39b716ab90de993230" candidate_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113e72b8e9b84b6bb8104d1992a65384" is_pre_election: true
I20260619 18:52:57.720316 17149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59bf26151ada4e39b716ab90de993230" candidate_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a477142e8548b59b6cfdadc914331f" is_pre_election: true
I20260619 18:52:57.720575 17008 raft_consensus.cc:2454] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ebdb0e48584abb835e39fd55d6a6c9 in term 0.
I20260619 18:52:57.720621 17149 raft_consensus.cc:2454] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ebdb0e48584abb835e39fd55d6a6c9 in term 0.
I20260619 18:52:57.721148 17221 leader_election.cc:302] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ebdb0e48584abb835e39fd55d6a6c9, 56a477142e8548b59b6cfdadc914331f; no voters:
I20260619 18:52:57.721537 17368 raft_consensus.cc:2781] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:52:57.721688 17368 raft_consensus.cc:482] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:52:57.721797 17368 raft_consensus.cc:3037] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:57.724982 17368 raft_consensus.cc:504] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.725899 17368 leader_election.cc:288] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [CANDIDATE]: Term 1 election: Requested vote from peers 113e72b8e9b84b6bb8104d1992a65384 (127.15.91.1:36551), 56a477142e8548b59b6cfdadc914331f (127.15.91.2:46551)
I20260619 18:52:57.726876 17008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59bf26151ada4e39b716ab90de993230" candidate_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113e72b8e9b84b6bb8104d1992a65384"
I20260619 18:52:57.726938 17149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59bf26151ada4e39b716ab90de993230" candidate_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a477142e8548b59b6cfdadc914331f"
I20260619 18:52:57.727165 17008 raft_consensus.cc:3037] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:57.727480 17149 raft_consensus.cc:3037] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:57.730485 17008 raft_consensus.cc:2454] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ebdb0e48584abb835e39fd55d6a6c9 in term 1.
I20260619 18:52:57.730551 17149 raft_consensus.cc:2454] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ebdb0e48584abb835e39fd55d6a6c9 in term 1.
I20260619 18:52:57.731262 17222 leader_election.cc:302] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113e72b8e9b84b6bb8104d1992a65384, 26ebdb0e48584abb835e39fd55d6a6c9; no voters:
I20260619 18:52:57.731974 17368 raft_consensus.cc:2781] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:57.732213 17368 raft_consensus.cc:686] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 LEADER]: Becoming Leader. State: Replica: 26ebdb0e48584abb835e39fd55d6a6c9, State: Running, Role: LEADER
I20260619 18:52:57.732551 17368 consensus_queue.cc:237] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } }
I20260619 18:52:57.734490 17389 tablet_replica.cc:439] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: TxnStatusTablet state changed. Reason: New leader 26ebdb0e48584abb835e39fd55d6a6c9. Latest consensus state: current_term: 1 leader_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } }
I20260619 18:52:57.734887 17389 tablet_replica.cc:442] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:52:57.735783 17402 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:52:57.736052 17402 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:52:57.736522 16851 catalog_manager.cc:5495] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26ebdb0e48584abb835e39fd55d6a6c9 (127.15.91.3). New cstate: current_term: 1 leader_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } health_report { overall_health: HEALTHY } } }
I20260619 18:52:57.769912 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.761923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.15.91.3:34813)
0619 18:52:57.763261 (+ 1338us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.763266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.763286 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:52:57.763378 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.763382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.763515 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.763730 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:57.764284 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.765331 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.767053 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.767616 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.767774 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.768432 (+ 658us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:57.769110 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:57.769290 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.769397 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.769489 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":1077,"threads_started":1}
I20260619 18:52:57.769889 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.761828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:34813 (local address 127.0.0.1:57380)
0619 18:52:57.762115 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:57.762138 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:57.762301 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:57.763763 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:57.763769 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:57.763788 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:52:57.764030 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.764041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.765497 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.765504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.766805 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.766824 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.767820 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.767828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.768072 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.768157 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:52:57.769249 (+ 1092us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:52:57.769285 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:52:57.769376 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 18:52:57.769565 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 18:52:57.776304 17008 raft_consensus.cc:1261] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384 [term 1 FOLLOWER]: Refusing update from remote peer 26ebdb0e48584abb835e39fd55d6a6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:52:57.776427 17149 raft_consensus.cc:1261] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f [term 1 FOLLOWER]: Refusing update from remote peer 26ebdb0e48584abb835e39fd55d6a6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:52:57.777522 17368 consensus_queue.cc:1035] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:52:57.778515 17368 consensus_queue.cc:1035] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:52:57.782749 17367 tablet_replica.cc:439] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384: TxnStatusTablet state changed. Reason: New leader 26ebdb0e48584abb835e39fd55d6a6c9. Latest consensus state: current_term: 1 leader_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } }
I20260619 18:52:57.783113 17367 tablet_replica.cc:442] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:52:57.785817 17406 mvcc.cc:205] Tried to move back new op lower bound from 7298642648160706560 to 7298642647995449344. Current Snapshot: MvccSnapshot[applied={T|T < 7298642648160706560}]
I20260619 18:52:57.788197 17367 tablet_replica.cc:439] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } }
I20260619 18:52:57.788491 17367 tablet_replica.cc:442] T 59bf26151ada4e39b716ab90de993230 P 113e72b8e9b84b6bb8104d1992a65384: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:52:57.796202 17364 tablet_replica.cc:439] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f: TxnStatusTablet state changed. Reason: New leader 26ebdb0e48584abb835e39fd55d6a6c9. Latest consensus state: current_term: 1 leader_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } }
I20260619 18:52:57.796715 17364 tablet_replica.cc:442] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:52:57.803623 17368 tablet_replica.cc:439] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } }
I20260619 18:52:57.803987 17368 tablet_replica.cc:442] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:52:57.806201 17389 tablet_replica.cc:439] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } }
I20260619 18:52:57.806500 17389 tablet_replica.cc:442] T 59bf26151ada4e39b716ab90de993230 P 26ebdb0e48584abb835e39fd55d6a6c9: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:52:57.807936 17364 tablet_replica.cc:439] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 } } peers { permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 } } peers { permanent_uuid: "26ebdb0e48584abb835e39fd55d6a6c9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 34813 } } }
I20260619 18:52:57.808405 17364 tablet_replica.cc:442] T 59bf26151ada4e39b716ab90de993230 P 56a477142e8548b59b6cfdadc914331f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:52:57.819495 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.812543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.15.91.3:34813)
0619 18:52:57.812759 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.812766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.813212 (+ 446us) server_negotiation.cc:407] Connection header received
0619 18:52:57.813358 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.813367 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.813475 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.813592 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:57.814041 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.815283 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.816934 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.817655 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.817821 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.818328 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:57.818485 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:57.818593 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.819055 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.819162 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:52:57.841153 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.833717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:39001 (local address 127.15.91.3:34813)
0619 18:52:57.833914 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.833919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.834471 (+ 552us) server_negotiation.cc:407] Connection header received
0619 18:52:57.834689 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.834695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.834816 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.834956 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:57.835512 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.836752 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.838557 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.839326 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.839540 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.840184 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:57.840362 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:57.840467 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.840624 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.840724 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:52:57.841172 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.833593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:34813 (local address 127.15.91.3:39001)
0619 18:52:57.834334 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:57.834352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:57.834527 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:57.834969 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:57.834973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:57.834984 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:52:57.835335 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.835345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.836965 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.836970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.838332 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:57.838355 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.839663 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.839670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:57.839837 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.839912 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:52:57.840493 (+ 581us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:52:57.840502 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:52:57.840573 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:52:57.840766 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":455,"threads_started":1}
I20260619 18:52:57.852427 17148 raft_consensus.cc:1261] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f [term 1 FOLLOWER]: Refusing update from remote peer 26ebdb0e48584abb835e39fd55d6a6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:52:57.852414 17008 raft_consensus.cc:1261] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384 [term 1 FOLLOWER]: Refusing update from remote peer 26ebdb0e48584abb835e39fd55d6a6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:52:57.853199 17389 consensus_queue.cc:1035] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56a477142e8548b59b6cfdadc914331f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 46551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:52:57.853578 17368 consensus_queue.cc:1035] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "113e72b8e9b84b6bb8104d1992a65384" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:52:57.943751 16850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:52:57.944047 16850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:52:57.948518 16850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d282e30c2c984eb79ab708d355ab8151: Sending DeleteTablet for 3 replicas of tablet 85389e4af36d4d368a5314cd343e61a6
I20260619 18:52:57.949848 16988 tablet_service.cc:1514] Processing DeleteTablet for tablet 85389e4af36d4d368a5314cd343e61a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:52:57 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:47110
I20260619 18:52:57.950129 17269 tablet_service.cc:1514] Processing DeleteTablet for tablet 85389e4af36d4d368a5314cd343e61a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:52:57 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:57364
I20260619 18:52:57.951004 17425 tablet_replica.cc:330] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384: stopping tablet replica
I20260619 18:52:57.951059 17129 tablet_service.cc:1514] Processing DeleteTablet for tablet 85389e4af36d4d368a5314cd343e61a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:52:57 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:51090
I20260619 18:52:57.951319 17425 raft_consensus.cc:2229] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:52:57.951741 17425 raft_consensus.cc:2258] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:52:57.953477 17425 ts_tablet_manager.cc:1905] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:52:57.959206 17427 tablet_replica.cc:330] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f: stopping tablet replica
I20260619 18:52:57.959470 17427 raft_consensus.cc:2229] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:52:57.959798 17427 raft_consensus.cc:2258] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:52:57.962659 17427 ts_tablet_manager.cc:1905] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:52:57.964316 17425 ts_tablet_manager.cc:1918] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:52:57.964548 17425 log.cc:1196] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/85389e4af36d4d368a5314cd343e61a6
I20260619 18:52:57.965193 17425 ts_tablet_manager.cc:1939] T 85389e4af36d4d368a5314cd343e61a6 P 113e72b8e9b84b6bb8104d1992a65384: Deleting consensus metadata
I20260619 18:52:57.969517 16839 catalog_manager.cc:4841] TS 113e72b8e9b84b6bb8104d1992a65384 (127.15.91.1:36551): tablet 85389e4af36d4d368a5314cd343e61a6 (table test-table [id=1dd89e5e65f444098780f7f4d6aa6a73]) successfully deleted
I20260619 18:52:57.972450 17427 ts_tablet_manager.cc:1918] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:52:57.972795 17427 log.cc:1196] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/85389e4af36d4d368a5314cd343e61a6
I20260619 18:52:57.973366 17427 ts_tablet_manager.cc:1939] T 85389e4af36d4d368a5314cd343e61a6 P 56a477142e8548b59b6cfdadc914331f: Deleting consensus metadata
I20260619 18:52:57.974776 16838 catalog_manager.cc:4841] TS 56a477142e8548b59b6cfdadc914331f (127.15.91.2:46551): tablet 85389e4af36d4d368a5314cd343e61a6 (table test-table [id=1dd89e5e65f444098780f7f4d6aa6a73]) successfully deleted
I20260619 18:52:57.976065 17426 tablet_replica.cc:330] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9: stopping tablet replica
I20260619 18:52:57.976382 17426 raft_consensus.cc:2229] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:52:57.977180 17426 raft_consensus.cc:2258] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:52:57.978458 17426 ts_tablet_manager.cc:1905] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:52:57 dist-test-slave-dm3f krb5kdc[16807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895176, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:52:57.988095 17426 ts_tablet_manager.cc:1918] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:52:57.988288 17426 log.cc:1196] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/85389e4af36d4d368a5314cd343e61a6
I20260619 18:52:57.988899 17426 ts_tablet_manager.cc:1939] T 85389e4af36d4d368a5314cd343e61a6 P 26ebdb0e48584abb835e39fd55d6a6c9: Deleting consensus metadata
I20260619 18:52:57.990492 16836 catalog_manager.cc:4841] TS 26ebdb0e48584abb835e39fd55d6a6c9 (127.15.91.3:34813): tablet 85389e4af36d4d368a5314cd343e61a6 (table test-table [id=1dd89e5e65f444098780f7f4d6aa6a73]) successfully deleted
I20260619 18:52:57.996014 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:57.966259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.15.91.1:36551)
0619 18:52:57.966477 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:57.966483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:57.974329 (+ 7846us) server_negotiation.cc:407] Connection header received
0619 18:52:57.974997 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:57.975007 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:57.975132 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:57.975255 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:57.978057 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.979269 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.981556 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:57.982242 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:57.982367 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:57.988530 (+ 6163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:57.988557 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:57.988584 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:57.988644 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:57.992478 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:57.993378 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:57.993392 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:57.993401 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:57.993573 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:57.994033 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:57.994038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:57.994043 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:57.994438 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:57.994819 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:57.995160 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 18:52:57.995408 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260619 18:52:57.996855 17054 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47122
I20260619 18:52:58.015511 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:58.000253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.15.91.62:44983)
0619 18:52:58.000447 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:58.000452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:58.000764 (+ 312us) server_negotiation.cc:407] Connection header received
0619 18:52:58.000915 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:58.000932 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:58.001041 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:58.001167 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:58.002585 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:58.003482 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:58.004848 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:58.005352 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:58.005517 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:58.008632 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:58.008638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:58.008642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:58.008686 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:58.012188 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:58.013137 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:58.013151 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:58.013154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:58.013317 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:58.013782 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:58.013787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:58.013792 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:58.014145 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:58.014468 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:58.014883 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 18:52:58.015156 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 18:52:58.016309 16850 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40552
I20260619 18:52:58.018878 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 16923
I20260619 18:52:58.047770 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:58.020371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.15.91.1:36551)
0619 18:52:58.020586 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:58.020593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:58.032614 (+ 12021us) server_negotiation.cc:407] Connection header received
0619 18:52:58.032875 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:58.032884 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:58.032980 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:58.033044 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:58.034506 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:58.035410 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:58.036808 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:58.037261 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:58.037441 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:58.040266 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:58.040272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:58.040276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:58.040323 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:58.044346 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:58.045327 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:58.045335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:58.045339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:58.045541 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:58.046005 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:58.046011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:58.046016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:58.046348 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:58.046676 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:58.047058 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 18:52:58.047337 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 18:52:58.048489 17054 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47136
W20260619 18:52:58.049176 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 16923: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:58.049387 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 16923
I20260619 18:52:58.068620 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 17063
Jun 19 18:52:58 dist-test-slave-dm3f krb5kdc[16807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895176, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:52:58.086110 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:58.069530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.15.91.2:46551)
0619 18:52:58.070072 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:58.070077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:58.070095 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:52:58.070180 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:58.070188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:58.070267 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:58.070393 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:58.071878 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:58.072659 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:58.074051 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:58.074499 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:58.074657 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:58.079415 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:58.079433 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:58.079451 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:58.079502 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:58.082916 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:58.083771 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:58.083780 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:58.083783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:58.083948 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:58.084386 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:58.084392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:58.084395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:58.084801 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:58.085135 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:58.085486 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 18:52:58.085737 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":308,"threads_started":1}
W20260619 18:52:58.086905 17189 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51116
W20260619 18:52:58.087675 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 17063: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:58.087883 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 17063
I20260619 18:52:58.110394 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 17203
Jun 19 18:52:58 dist-test-slave-dm3f krb5kdc[16807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895176, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:52:58.130254 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:58.111286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57402 (local address 127.15.91.3:34813)
0619 18:52:58.111632 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:58.111639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:58.111662 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:52:58.111753 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:58.111761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:58.111885 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:58.112021 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:58.113322 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:58.114454 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:58.116025 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:58.116879 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:58.117070 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:58.121852 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:58.121874 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:58.121903 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:58.121965 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:58.126628 (+ 4663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:58.127699 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:58.127709 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:58.127712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:58.127942 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:58.128385 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:58.128390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:58.128393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:58.128916 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:58.129314 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:58.129556 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 18:52:58.129834 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
W20260619 18:52:58.131078 17334 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57402
W20260619 18:52:58.131891 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 17203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:58.132078 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 17203
I20260619 18:52:58.153614 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 16820
I20260619 18:52:58.155001 16892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:58.272883 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 16820
2026-06-19T18:52:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5038 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:58 dist-test-slave-dm3f krb5kdc[17437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:58 dist-test-slave-dm3f krb5kdc[17437](info): set up 2 sockets
Jun 19 18:52:58 dist-test-slave-dm3f krb5kdc[17437](info): commencing operation
krb5kdc: starting...
W20260619 18:52:59.375128 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:52:59 dist-test-slave-dm3f krb5kdc[17437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:52:59.582476 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:33063
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:33063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:59.957154 17450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:59.957497 17450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:59.957592 17450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:59.965482 17450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:59.965622 17450 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:52:59.965663 17450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:59.965718 17450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:59.965782 17450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:59.965847 17450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:59.977073 17450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:33063
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:33063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:59.978423 17450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:59.979987 17450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:59.993402 17455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:59.993358 17458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:59.993319 17456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:59.994786 17450 server_base.cc:1034] running on GCE node
I20260619 18:52:59.995756 17450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:59.997566 17450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:59.998870 17450 hybrid_clock.cc:648] HybridClock initialized: now 1781895179998778 us; error 77 us; skew 500 ppm
I20260619 18:52:59.999286 17450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:59 dist-test-slave-dm3f krb5kdc[17437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895179, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:00.003126 17450 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:53:00.005023 17450 webserver.cc:460] Webserver started at http://127.15.91.62:39749/ using document root <none> and password file <none>
I20260619 18:53:00.005946 17450 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:00.006130 17450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:00.006510 17450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:00.010438 17450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "6303b1da75ec4564be2c76382ed15b9e"
format_stamp: "Formatted at 2026-06-19 18:53:00 on dist-test-slave-dm3f"
server_key: "d93751b798d4a5160dc422a6553243d1"
server_key_iv: "3b7a675faf426290987e7be0756db334"
server_key_version: "encryptionkey@0"
I20260619 18:53:00.011619 17450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "6303b1da75ec4564be2c76382ed15b9e"
format_stamp: "Formatted at 2026-06-19 18:53:00 on dist-test-slave-dm3f"
server_key: "d93751b798d4a5160dc422a6553243d1"
server_key_iv: "3b7a675faf426290987e7be0756db334"
server_key_version: "encryptionkey@0"
I20260619 18:53:00.018641 17450 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 18:53:00.023322 17465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:00.025228 17450 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:53:00.025503 17450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "6303b1da75ec4564be2c76382ed15b9e"
format_stamp: "Formatted at 2026-06-19 18:53:00 on dist-test-slave-dm3f"
server_key: "d93751b798d4a5160dc422a6553243d1"
server_key_iv: "3b7a675faf426290987e7be0756db334"
server_key_version: "encryptionkey@0"
I20260619 18:53:00.025817 17450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:00.041757 17450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:00.047358 17450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:00.047748 17450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:00.072490 17450 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:33063
I20260619 18:53:00.072497 17526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:33063 every 8 connection(s)
I20260619 18:53:00.074116 17450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:00.078418 17527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:00.083897 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 17450
I20260619 18:53:00.084136 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:00.084797 15724 external_mini_cluster.cc:1428] Setting key f31d7b9db2fe8f3c27ee088c7f1869fb
I20260619 18:53:00.091980 17527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e: Bootstrap starting.
I20260619 18:53:00.096318 17527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:00.097679 17527 log.cc:824] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:00.101084 17527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e: No bootstrap required, opened a new log
Jun 19 18:53:00 dist-test-slave-dm3f krb5kdc[17437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:00.112480 17527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6303b1da75ec4564be2c76382ed15b9e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33063 } }
I20260619 18:53:00.113034 17527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:00.113149 17527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6303b1da75ec4564be2c76382ed15b9e, State: Initialized, Role: FOLLOWER
I20260619 18:53:00.113579 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:00.087175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.15.91.62:33063)
0619 18:53:00.088131 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:00.088155 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:00.088216 (+ 61us) server_negotiation.cc:407] Connection header received
0619 18:53:00.090848 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:00.090884 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:00.091500 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:00.092038 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:00.093613 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:00.095770 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:00.097281 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:00.098525 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:00.098684 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:00.103411 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:00.103430 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:00.103454 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:00.103520 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:00.107951 (+ 4431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:00.108869 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:00.108880 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:00.108891 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:00.109174 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:00.109593 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:00.109601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:00.109611 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:00.110548 (+ 937us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:00.110946 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:00.111426 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 18:53:00.111958 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":382,"threads_started":1}
I20260619 18:53:00.113986 17527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6303b1da75ec4564be2c76382ed15b9e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33063 } }
I20260619 18:53:00.114320 17527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:00.114538 17527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:00.114770 17527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:00.117560 17527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6303b1da75ec4564be2c76382ed15b9e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33063 } }
I20260619 18:53:00.118220 17527 leader_election.cc:302] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6303b1da75ec4564be2c76382ed15b9e; no voters:
I20260619 18:53:00.119028 17527 leader_election.cc:288] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:00.119150 17532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:00.120007 17532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [term 1 LEADER]: Becoming Leader. State: Replica: 6303b1da75ec4564be2c76382ed15b9e, State: Running, Role: LEADER
I20260619 18:53:00.120826 17532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6303b1da75ec4564be2c76382ed15b9e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33063 } }
I20260619 18:53:00.122191 17527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:00.124007 17533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6303b1da75ec4564be2c76382ed15b9e. Latest consensus state: current_term: 1 leader_uuid: "6303b1da75ec4564be2c76382ed15b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6303b1da75ec4564be2c76382ed15b9e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33063 } } }
I20260619 18:53:00.124564 17533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:00.127393 17534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6303b1da75ec4564be2c76382ed15b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6303b1da75ec4564be2c76382ed15b9e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33063 } } }
I20260619 18:53:00.127843 17534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:00.130290 17538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:00.136655 17538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:00.146803 17538 catalog_manager.cc:1347] Generated new cluster ID: 93ab109a4df149f0a2c8ae19de7e92c5
I20260619 18:53:00.146955 17538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:00.176110 17538 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:00.178259 17538 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:00.191555 17538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e: Generated new TSK 0
I20260619 18:53:00.192782 17538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:53:00.245998 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:33063
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:00.622417 17553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:00.622803 17553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:00.622938 17553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:00.631606 17553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:00.631752 17553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:00.631888 17553 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:53:00.631994 17553 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:53:00.632129 17553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:53:00.642797 17553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.91.62:33063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:00.644253 17553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:00.645750 17553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:00.660681 17561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:00.660778 17558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:00.660754 17559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:00.662159 17553 server_base.cc:1034] running on GCE node
I20260619 18:53:00.663087 17553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:00.664250 17553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:00.665551 17553 hybrid_clock.cc:648] HybridClock initialized: now 1781895180665485 us; error 62 us; skew 500 ppm
I20260619 18:53:00.665975 17553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:00 dist-test-slave-dm3f krb5kdc[17437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895180, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:00.672956 17553 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:53:00.674844 17553 webserver.cc:460] Webserver started at http://127.15.91.1:33229/ using document root <none> and password file <none>
I20260619 18:53:00.675625 17553 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:00.675770 17553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:00.676137 17553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:00.679524 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "032e4366417c422bb64edde0a27f6ab2"
format_stamp: "Formatted at 2026-06-19 18:53:00 on dist-test-slave-dm3f"
server_key: "8d5b248b9e0c81c42b27945d15d5b6cd"
server_key_iv: "7b03be457df15948526222618ecf8ed5"
server_key_version: "encryptionkey@0"
I20260619 18:53:00.680449 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "032e4366417c422bb64edde0a27f6ab2"
format_stamp: "Formatted at 2026-06-19 18:53:00 on dist-test-slave-dm3f"
server_key: "8d5b248b9e0c81c42b27945d15d5b6cd"
server_key_iv: "7b03be457df15948526222618ecf8ed5"
server_key_version: "encryptionkey@0"
I20260619 18:53:00.686570 17553 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:53:00.691021 17568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:00.692866 17553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:53:00.693126 17553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "032e4366417c422bb64edde0a27f6ab2"
format_stamp: "Formatted at 2026-06-19 18:53:00 on dist-test-slave-dm3f"
server_key: "8d5b248b9e0c81c42b27945d15d5b6cd"
server_key_iv: "7b03be457df15948526222618ecf8ed5"
server_key_version: "encryptionkey@0"
I20260619 18:53:00.693399 17553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:00.721015 17553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:00.726939 17553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:00.727458 17553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:00.731717 17553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:00.731892 17553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:00.732029 17553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:00.732134 17553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:53:00 dist-test-slave-dm3f krb5kdc[17437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895180, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:00.763515 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:00.733354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:33063 (local address 127.15.91.1:36643)
0619 18:53:00.734343 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:00.734399 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:00.736819 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:00.738252 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:00.738276 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:00.739269 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:00.740439 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:00.740472 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:00.741868 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:00.741873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:00.744259 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:00.744279 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:00.745496 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:00.745512 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:00.745978 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:00.747462 (+ 1484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:00.747512 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:00.751800 (+ 4288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:00.756947 (+ 5147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:00.756962 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:00.756987 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:00.757990 (+ 1003us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:00.758506 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:00.758515 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:00.758521 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:00.758989 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:00.759657 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:00.759666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:00.760251 (+ 585us) client_negotiation.cc:769] Sending connection context
0619 18:53:00.760733 (+ 482us) client_negotiation.cc:240] Negotiation successful
0619 18:53:00.761437 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":421,"threads_started":1}
I20260619 18:53:00.764341 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:00.732473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:36643 (local address 127.15.91.62:33063)
0619 18:53:00.733209 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:00.733213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:00.734564 (+ 1351us) server_negotiation.cc:407] Connection header received
0619 18:53:00.737302 (+ 2738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:00.737312 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:00.737453 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:00.737554 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:00.740714 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:00.741651 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:00.744484 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:00.745133 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:00.745315 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:00.752262 (+ 6947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:00.752274 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:00.752282 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:00.752503 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:00.756647 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:00.758204 (+ 1557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:00.758209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:00.758212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:00.758362 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:00.759161 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:00.759168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:00.759173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:00.759500 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:00.759684 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:00.763607 (+ 3923us) server_negotiation.cc:299] Negotiation successful
0619 18:53:00.763937 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":491,"threads_started":1}
I20260619 18:53:00.792426 17553 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:45737
I20260619 18:53:00.792438 17688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:45737 every 8 connection(s)
I20260619 18:53:00.794091 17553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:53:00.797233 17689 heartbeater.cc:347] Connected to a master server at 127.15.91.62:33063
I20260619 18:53:00.797576 17689 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:00.798604 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:00.779892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:44345 (local address 127.15.91.62:33063)
0619 18:53:00.780107 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:00.780114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:00.780804 (+ 690us) server_negotiation.cc:407] Connection header received
0619 18:53:00.781162 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:00.781168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:00.781261 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:00.781384 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:00.782744 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:00.783812 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:00.786354 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:00.787045 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:00.787164 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:00.790344 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:00.790350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:00.790354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:00.790393 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:00.794483 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:00.795826 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:00.795833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:00.795837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:00.796053 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:00.796576 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:00.796581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:00.796583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:00.797073 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:00.797243 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:00.797942 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 18:53:00.798158 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:53:00.798610 17689 heartbeater.cc:511] Master 127.15.91.62:33063 requested a full tablet report, sending...
I20260619 18:53:00.798830 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 17553
I20260619 18:53:00.798868 17663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:00.779890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:33063 (local address 127.15.91.1:44345)
0619 18:53:00.780656 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:00.780679 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:00.780961 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:00.781446 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:00.781452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:00.782026 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:00.782430 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:00.782454 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:00.784114 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:00.784121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:00.786021 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:00.786059 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:00.787554 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:00.787565 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:00.787831 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:00.788782 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:00.788817 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:00.790047 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:00.794768 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:00.794780 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:00.794788 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:00.795557 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:00.796220 (+ 663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:00.796225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:00.796228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:00.796394 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:00.797268 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:00.797275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:00.797578 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 18:53:00.797781 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 18:53:00.798192 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":418,"threads_started":1}
I20260619 18:53:00.799057 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:53:00.799815 15724 external_mini_cluster.cc:1428] Setting key a7710ea1b426abee010dbe773fff9ce7
I20260619 18:53:00.803282 17480 ts_manager.cc:194] Registered new tserver with Master: 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737)
I20260619 18:53:00.805639 17480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:36643
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:53:00.851504 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:33063
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:01.207976 17693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:01.208310 17693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:01.208426 17693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:01.217033 17693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:01.217190 17693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:01.217288 17693 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:53:01.217342 17693 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:53:01.217415 17693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:53:01.229549 17693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.91.62:33063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:01.231189 17693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:01.232744 17693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:01.247829 17699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:01.247892 17701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:01.247931 17698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:01.248855 17693 server_base.cc:1034] running on GCE node
I20260619 18:53:01.249542 17693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:01.250696 17693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:01.252031 17693 hybrid_clock.cc:648] HybridClock initialized: now 1781895181251950 us; error 78 us; skew 500 ppm
I20260619 18:53:01.252449 17693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:01 dist-test-slave-dm3f krb5kdc[17437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895181, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:01.256243 17693 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:53:01.257987 17693 webserver.cc:460] Webserver started at http://127.15.91.2:32801/ using document root <none> and password file <none>
I20260619 18:53:01.258828 17693 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:01.258965 17693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:01.259250 17693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:01.262437 17693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "05e6c1321811408b933914fb7a98c76d"
format_stamp: "Formatted at 2026-06-19 18:53:01 on dist-test-slave-dm3f"
server_key: "1aa38cf00273d1d9ab201d5817b839be"
server_key_iv: "3ae9a9b894e729fe6a62353499165952"
server_key_version: "encryptionkey@0"
I20260619 18:53:01.263402 17693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "05e6c1321811408b933914fb7a98c76d"
format_stamp: "Formatted at 2026-06-19 18:53:01 on dist-test-slave-dm3f"
server_key: "1aa38cf00273d1d9ab201d5817b839be"
server_key_iv: "3ae9a9b894e729fe6a62353499165952"
server_key_version: "encryptionkey@0"
I20260619 18:53:01.269776 17693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 18:53:01.274430 17708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:01.276291 17693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260619 18:53:01.276542 17693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "05e6c1321811408b933914fb7a98c76d"
format_stamp: "Formatted at 2026-06-19 18:53:01 on dist-test-slave-dm3f"
server_key: "1aa38cf00273d1d9ab201d5817b839be"
server_key_iv: "3ae9a9b894e729fe6a62353499165952"
server_key_version: "encryptionkey@0"
I20260619 18:53:01.276760 17693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:01.293617 17693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:01.299062 17693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:01.299477 17693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:01.302575 17693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:01.302773 17693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:01.302949 17693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:01.303050 17693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:53:01 dist-test-slave-dm3f krb5kdc[17437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895181, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:01.331773 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:01.303569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:43441 (local address 127.15.91.62:33063)
0619 18:53:01.304061 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:01.304065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:01.305759 (+ 1694us) server_negotiation.cc:407] Connection header received
0619 18:53:01.308380 (+ 2621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:01.308386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:01.308508 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:01.308651 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:01.311631 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.312837 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.315853 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.316558 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.316855 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:01.322786 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:01.322795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:01.322800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:01.322854 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:01.327007 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:01.328540 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:01.328548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:01.328552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:01.328797 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:01.329435 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:01.329438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:01.329441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:01.329848 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:01.330050 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:01.331053 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0619 18:53:01.331344 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":266,"threads_started":1}
I20260619 18:53:01.333410 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:01.304480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:33063 (local address 127.15.91.2:43441)
0619 18:53:01.305537 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:01.305598 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:01.307999 (+ 2401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:01.309018 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:01.309033 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:01.310070 (+ 1037us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:01.311330 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:01.311373 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.313088 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.313093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:01.315616 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:01.315639 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.317046 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.317056 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:01.317426 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:01.318451 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:01.318498 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:01.322349 (+ 3851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:01.327295 (+ 4946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:01.327317 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:01.327346 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:01.328342 (+ 996us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:01.328990 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:01.328996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:01.329000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:01.329285 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:01.330028 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:01.330038 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:01.330469 (+ 431us) client_negotiation.cc:769] Sending connection context
0619 18:53:01.330913 (+ 444us) client_negotiation.cc:240] Negotiation successful
0619 18:53:01.331615 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":476,"threads_started":1}
I20260619 18:53:01.356545 17693 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:34755
I20260619 18:53:01.356595 17828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:34755 every 8 connection(s)
I20260619 18:53:01.358007 17693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:53:01.359238 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:01.342357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:34539 (local address 127.15.91.62:33063)
0619 18:53:01.342542 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:01.342547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:01.343595 (+ 1048us) server_negotiation.cc:407] Connection header received
0619 18:53:01.343852 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:01.343858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:01.343947 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:01.344067 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:01.345527 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.346589 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.348907 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.349372 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.349556 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:01.352190 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:01.352196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:01.352199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:01.352236 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:01.355919 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:01.356880 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:01.356884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:01.356887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:01.357034 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:01.357470 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:01.357481 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:01.357483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:01.357843 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:01.358023 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:01.358539 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 18:53:01.358866 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:53:01.359583 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:01.342483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:33063 (local address 127.15.91.2:34539)
0619 18:53:01.343440 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:01.343460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:01.343700 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:01.344106 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:01.344110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:01.344842 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:01.345323 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:01.345338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.346918 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.346925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:01.348683 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:01.348706 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.349621 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.349629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:01.349806 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:01.350657 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:01.350677 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:01.351883 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:01.356113 (+ 4230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:01.356119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:01.356124 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:01.356715 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:01.357170 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:01.357176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:01.357180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:01.357328 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:01.357998 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:01.358003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:01.358228 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 18:53:01.358400 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 18:53:01.359002 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":569,"threads_started":1}
I20260619 18:53:01.361248 17829 heartbeater.cc:347] Connected to a master server at 127.15.91.62:33063
I20260619 18:53:01.361742 17829 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:01.362489 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 17693
I20260619 18:53:01.362673 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:53:01.363073 17829 heartbeater.cc:511] Master 127.15.91.62:33063 requested a full tablet report, sending...
I20260619 18:53:01.363317 15724 external_mini_cluster.cc:1428] Setting key 3089a6da2859fbf3810a37723d921394
I20260619 18:53:01.365272 17480 ts_manager.cc:194] Registered new tserver with Master: 05e6c1321811408b933914fb7a98c76d (127.15.91.2:34755)
I20260619 18:53:01.366438 17480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:43441
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:53:01.410681 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:33063
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:01.773702 17833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:01.774021 17833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:01.774199 17833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:01.782473 17833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:01.782616 17833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:01.782804 17833 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:53:01.782887 17833 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:53:01.783031 17833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:53:01.793671 17833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.91.62:33063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:01.795109 17833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:01.796690 17833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:53:01.809777 17689 heartbeater.cc:503] Master 127.15.91.62:33063 was elected leader, sending a full tablet report...
W20260619 18:53:01.812492 17839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:01.812529 17841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:01.812736 17838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:01.813825 17833 server_base.cc:1034] running on GCE node
I20260619 18:53:01.814535 17833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:01.815670 17833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:01.816946 17833 hybrid_clock.cc:648] HybridClock initialized: now 1781895181816889 us; error 51 us; skew 500 ppm
I20260619 18:53:01.817323 17833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:01 dist-test-slave-dm3f krb5kdc[17437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895181, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:01.820992 17833 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:53:01.822851 17833 webserver.cc:460] Webserver started at http://127.15.91.3:41505/ using document root <none> and password file <none>
I20260619 18:53:01.823671 17833 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:01.823792 17833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:01.824093 17833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:01.827479 17833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0"
format_stamp: "Formatted at 2026-06-19 18:53:01 on dist-test-slave-dm3f"
server_key: "c1021a0fa048fe914d5a0e25b443ff09"
server_key_iv: "97e623d513cf4dd77e9ed1cfaec92c98"
server_key_version: "encryptionkey@0"
I20260619 18:53:01.828392 17833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0"
format_stamp: "Formatted at 2026-06-19 18:53:01 on dist-test-slave-dm3f"
server_key: "c1021a0fa048fe914d5a0e25b443ff09"
server_key_iv: "97e623d513cf4dd77e9ed1cfaec92c98"
server_key_version: "encryptionkey@0"
I20260619 18:53:01.834867 17833 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:53:01.839479 17848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:01.841271 17833 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:53:01.841524 17833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0"
format_stamp: "Formatted at 2026-06-19 18:53:01 on dist-test-slave-dm3f"
server_key: "c1021a0fa048fe914d5a0e25b443ff09"
server_key_iv: "97e623d513cf4dd77e9ed1cfaec92c98"
server_key_version: "encryptionkey@0"
I20260619 18:53:01.841759 17833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:01.868285 17833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:01.875007 17833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:01.875466 17833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:01.879328 17833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:01.879532 17833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:01.879690 17833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:01.879778 17833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:53:01 dist-test-slave-dm3f krb5kdc[17437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895181, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:01.911803 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:01.880309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:47469 (local address 127.15.91.62:33063)
0619 18:53:01.881431 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:01.881437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:01.883066 (+ 1629us) server_negotiation.cc:407] Connection header received
0619 18:53:01.885474 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:01.885481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:01.885615 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:01.885766 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:01.890069 (+ 4303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.891313 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.896136 (+ 4823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.896781 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.896958 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:01.902547 (+ 5589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:01.902558 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:01.902568 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:01.902635 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:01.907227 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:01.908554 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:01.908565 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:01.908572 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:01.908819 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:01.909349 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:01.909357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:01.909363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:01.909879 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:01.910087 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:01.910940 (+ 853us) server_negotiation.cc:299] Negotiation successful
0619 18:53:01.911246 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":772,"threads_started":1}
I20260619 18:53:01.913544 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:01.881461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:33063 (local address 127.15.91.3:47469)
0619 18:53:01.882847 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:01.882904 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:01.885075 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:01.886421 (+ 1346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:01.886443 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:01.888057 (+ 1614us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:01.889677 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:01.889730 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.891653 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.891662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:01.895766 (+ 4104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:01.895801 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.897404 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.897417 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:01.897777 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:01.898892 (+ 1115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:01.898925 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:01.902132 (+ 3207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:01.907482 (+ 5350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:01.907495 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:01.907530 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:01.908329 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:01.908976 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:01.908979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:01.908982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:01.909209 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:01.910046 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:01.910053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:01.910453 (+ 400us) client_negotiation.cc:769] Sending connection context
0619 18:53:01.910709 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 18:53:01.911545 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":646,"threads_started":1}
I20260619 18:53:01.939263 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:01.922549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:45089 (local address 127.15.91.62:33063)
0619 18:53:01.922872 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:01.922882 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:01.923867 (+ 985us) server_negotiation.cc:407] Connection header received
0619 18:53:01.924126 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:01.924130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:01.924203 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:01.924302 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:01.926073 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.926943 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.929186 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.929636 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.929797 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:01.932339 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:01.932344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:01.932348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:01.932385 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:01.935489 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:01.936703 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:01.936711 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:01.936715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:01.936952 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:01.937387 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:01.937392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:01.937396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:01.937721 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:01.937895 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:01.938529 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 18:53:01.938819 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:53:01.940480 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:01.922554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:33063 (local address 127.15.91.3:45089)
0619 18:53:01.923696 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:01.923733 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:01.923979 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:01.924393 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:01.924401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:01.925298 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:01.925815 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:01.925850 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.927127 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.927133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:01.928969 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:01.928989 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:01.929892 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:01.929899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:01.930085 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:01.930919 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:01.930949 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:01.932073 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:01.935751 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:01.935756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:01.935759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:01.936521 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:01.937084 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:01.937091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:01.937099 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:01.937249 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:01.937858 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:01.937863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:01.938119 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 18:53:01.939488 (+ 1369us) client_negotiation.cc:240] Negotiation successful
0619 18:53:01.939879 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":616,"threads_started":1}
I20260619 18:53:01.949354 17833 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:37759
I20260619 18:53:01.949354 17968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:37759 every 8 connection(s)
I20260619 18:53:01.950865 17833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:53:01.953269 17969 heartbeater.cc:347] Connected to a master server at 127.15.91.62:33063
I20260619 18:53:01.953523 17969 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:01.954231 17969 heartbeater.cc:511] Master 127.15.91.62:33063 requested a full tablet report, sending...
I20260619 18:53:01.956060 17480 ts_manager.cc:194] Registered new tserver with Master: 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759)
I20260619 18:53:01.956916 17480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:47469
I20260619 18:53:01.959487 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 17833
I20260619 18:53:01.959702 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:53:01.960157 15724 external_mini_cluster.cc:1428] Setting key eb2830258a62d4bb6770240f9e69d523
I20260619 18:53:01.964018 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:53:01 dist-test-slave-dm3f krb5kdc[17437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:53:02 dist-test-slave-dm3f krb5kdc[17437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:02.013931 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:01.999077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.15.91.62:33063)
0619 18:53:01.999258 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:01.999263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:01.999690 (+ 427us) server_negotiation.cc:407] Connection header received
0619 18:53:01.999841 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:01.999845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:01.999949 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:02.000048 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:02.001561 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.002303 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.003711 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.004192 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.004309 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.008593 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:02.008599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:02.008603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:02.008646 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:02.011403 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:02.012154 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:02.012158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:02.012161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:02.012301 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:02.012667 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:02.012670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:02.012672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:02.012949 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:02.013190 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:02.013480 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 18:53:02.013674 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:53:02.018882 17480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:53:02.026265 17480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:02.054168 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.040629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.15.91.3:37759)
0619 18:53:02.043046 (+ 2417us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:02.043055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:02.043097 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:53:02.043218 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:02.043228 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:02.043560 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:02.043775 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:02.045044 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.046787 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.050260 (+ 3473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.052023 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.052161 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.052420 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:02.053559 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0619 18:53:02.053770 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2212,"thread_start_us":2001,"threads_started":1}
I20260619 18:53:02.054168 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.040323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:37759 (local address 127.0.0.1:56340)
0619 18:53:02.041492 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:02.041545 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:02.041738 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:02.044204 (+ 2466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:02.044213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:02.044247 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:02.044778 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.044793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.046997 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.047005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.049995 (+ 2990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.050014 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.052663 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.052669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.053117 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.053284 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 18:53:02.053459 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 18:53:02.053687 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":179,"thread_start_us":970,"threads_started":1}
I20260619 18:53:02.054425 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.041106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.15.91.2:34755)
0619 18:53:02.042430 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:02.042441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:02.043333 (+ 892us) server_negotiation.cc:407] Connection header received
0619 18:53:02.043663 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:02.043676 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:02.043907 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:02.044075 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:02.045000 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.046793 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.050449 (+ 3656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.052013 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.052160 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.052411 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:02.053571 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0619 18:53:02.053761 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":1025,"threads_started":1}
I20260619 18:53:02.055359 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.040531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:34755 (local address 127.0.0.1:50430)
0619 18:53:02.043216 (+ 2685us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:02.043236 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:02.043461 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:02.044323 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:02.044329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:02.044353 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:02.044814 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.044829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.047041 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.047048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.050210 (+ 3162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.050225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.052604 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.052615 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.053117 (+ 502us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.053284 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 18:53:02.054812 (+ 1528us) client_negotiation.cc:240] Negotiation successful
0619 18:53:02.054982 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2548,"mutex_wait_us":194,"thread_start_us":2412,"threads_started":1}
I20260619 18:53:02.057281 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.042445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.15.91.1:45737)
0619 18:53:02.049159 (+ 6714us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:02.049169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:02.049208 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:53:02.049303 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:02.049309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:02.049536 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:02.049791 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:02.050567 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.051969 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.054124 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.056080 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.056365 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.056612 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:02.056818 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 18:53:02.056983 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6556,"thread_start_us":6419,"threads_started":1}
I20260619 18:53:02.057337 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.042173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:45737 (local address 127.0.0.1:51060)
0619 18:53:02.048307 (+ 6134us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:02.048341 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:02.048577 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:02.049778 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:02.049784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:02.049817 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:02.050343 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.050359 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.052146 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.052153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.053898 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.053917 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.056416 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.056423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.056601 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.056644 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 18:53:02.056772 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 18:53:02.056922 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5908,"mutex_wait_us":38,"thread_start_us":5666,"threads_started":1}
I20260619 18:53:02.059154 17899 tablet_service.cc:1467] Processing CreateTablet for tablet 44127153a83a4bba83d00fb2f6629ffe (DEFAULT_TABLE table=test-table [id=a23eba9e473c4931a00d66dc95101de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:02.059154 17759 tablet_service.cc:1467] Processing CreateTablet for tablet 44127153a83a4bba83d00fb2f6629ffe (DEFAULT_TABLE table=test-table [id=a23eba9e473c4931a00d66dc95101de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:02.060774 17899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44127153a83a4bba83d00fb2f6629ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:02.060772 17759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44127153a83a4bba83d00fb2f6629ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:02.062561 17619 tablet_service.cc:1467] Processing CreateTablet for tablet 44127153a83a4bba83d00fb2f6629ffe (DEFAULT_TABLE table=test-table [id=a23eba9e473c4931a00d66dc95101de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:02.064003 17619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44127153a83a4bba83d00fb2f6629ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:02.084729 17990 tablet_bootstrap.cc:492] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: Bootstrap starting.
I20260619 18:53:02.090409 17990 tablet_bootstrap.cc:654] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:02.091717 17988 tablet_bootstrap.cc:492] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: Bootstrap starting.
I20260619 18:53:02.092897 17989 tablet_bootstrap.cc:492] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: Bootstrap starting.
I20260619 18:53:02.097648 17989 tablet_bootstrap.cc:654] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:02.098927 17988 tablet_bootstrap.cc:654] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:02.099416 17989 log.cc:824] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:02.100471 17988 log.cc:824] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:02.102583 17990 log.cc:824] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:02.104017 17989 tablet_bootstrap.cc:492] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: No bootstrap required, opened a new log
I20260619 18:53:02.104398 17989 ts_tablet_manager.cc:1397] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260619 18:53:02.104671 17988 tablet_bootstrap.cc:492] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: No bootstrap required, opened a new log
I20260619 18:53:02.104960 17988 ts_tablet_manager.cc:1397] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.007s
I20260619 18:53:02.106074 17990 tablet_bootstrap.cc:492] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: No bootstrap required, opened a new log
I20260619 18:53:02.106377 17990 ts_tablet_manager.cc:1397] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.004s
I20260619 18:53:02.112771 17988 raft_consensus.cc:348] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } }
I20260619 18:53:02.113277 17988 raft_consensus.cc:374] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:02.113386 17988 raft_consensus.cc:729] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032e4366417c422bb64edde0a27f6ab2, State: Initialized, Role: FOLLOWER
I20260619 18:53:02.114298 17988 consensus_queue.cc:260] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } }
I20260619 18:53:02.117012 17988 ts_tablet_manager.cc:1428] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260619 18:53:02.117235 17989 raft_consensus.cc:348] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } }
I20260619 18:53:02.117882 17989 raft_consensus.cc:374] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:02.118024 17989 raft_consensus.cc:729] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0bb2ae4bea47b59e5190f33a09d5b0, State: Initialized, Role: FOLLOWER
I20260619 18:53:02.118237 17990 raft_consensus.cc:348] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } }
I20260619 18:53:02.118824 17990 raft_consensus.cc:374] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:02.118974 17990 raft_consensus.cc:729] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e6c1321811408b933914fb7a98c76d, State: Initialized, Role: FOLLOWER
I20260619 18:53:02.119082 17989 consensus_queue.cc:260] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } }
I20260619 18:53:02.119796 17990 consensus_queue.cc:260] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } }
I20260619 18:53:02.120472 17969 heartbeater.cc:503] Master 127.15.91.62:33063 was elected leader, sending a full tablet report...
I20260619 18:53:02.121510 17989 ts_tablet_manager.cc:1428] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260619 18:53:02.123234 17829 heartbeater.cc:503] Master 127.15.91.62:33063 was elected leader, sending a full tablet report...
I20260619 18:53:02.124210 17990 ts_tablet_manager.cc:1428] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: Time spent starting tablet: real 0.018s user 0.005s sys 0.014s
W20260619 18:53:02.202800 17970 tablet.cc:2369] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:53:02.220234 17996 raft_consensus.cc:482] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:02.220557 17996 raft_consensus.cc:504] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } }
I20260619 18:53:02.222434 17996 leader_election.cc:288] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759), 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737)
I20260619 18:53:02.232667 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.223067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:39737 (local address 127.15.91.1:45737)
0619 18:53:02.223289 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:02.223297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:02.223868 (+ 571us) server_negotiation.cc:407] Connection header received
0619 18:53:02.224119 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:02.224124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:02.224216 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:02.224332 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:02.225202 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.226363 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.229838 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.231252 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.231425 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.231556 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:02.232216 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 18:53:02.232376 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:53:02.232667 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.222805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:38215 (local address 127.15.91.3:37759)
0619 18:53:02.223037 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:02.223043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:02.223983 (+ 940us) server_negotiation.cc:407] Connection header received
0619 18:53:02.224118 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:02.224124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:02.224216 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:02.224332 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:02.225204 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.226363 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.229845 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.231252 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.231425 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.231548 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:02.232238 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 18:53:02.232382 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:53:02.232928 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.222848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:37759 (local address 127.15.91.2:38215)
0619 18:53:02.223864 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:02.223880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:02.223983 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:02.224378 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:02.224383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:02.224448 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:02.225032 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.225045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.226570 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.226576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.229546 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.229576 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.231593 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.231602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.231961 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.232029 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 18:53:02.232197 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 18:53:02.232327 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":253,"spinlock_wait_cycles":38016,"thread_start_us":674,"threads_started":1}
I20260619 18:53:02.232969 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.222702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:45737 (local address 127.15.91.2:39737)
0619 18:53:02.223714 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:02.223740 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:02.223942 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:02.224378 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:02.224383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:02.224413 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:02.224937 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.224961 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.226570 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.226576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.229546 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.229576 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.231593 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.231602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.231961 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.232035 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 18:53:02.232157 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 18:53:02.232327 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":340,"spinlock_wait_cycles":1408,"thread_start_us":509,"threads_started":1}
I20260619 18:53:02.234318 17639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44127153a83a4bba83d00fb2f6629ffe" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032e4366417c422bb64edde0a27f6ab2" is_pre_election: true
I20260619 18:53:02.234318 17919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44127153a83a4bba83d00fb2f6629ffe" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" is_pre_election: true
I20260619 18:53:02.235018 17639 raft_consensus.cc:2454] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 0.
I20260619 18:53:02.235018 17919 raft_consensus.cc:2454] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 0.
I20260619 18:53:02.235914 17712 leader_election.cc:302] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 032e4366417c422bb64edde0a27f6ab2, 05e6c1321811408b933914fb7a98c76d, 8f0bb2ae4bea47b59e5190f33a09d5b0; no voters:
I20260619 18:53:02.236578 17996 raft_consensus.cc:2781] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:02.236754 17996 raft_consensus.cc:482] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:02.236836 17996 raft_consensus.cc:3037] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:02.239344 17996 raft_consensus.cc:504] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } }
I20260619 18:53:02.240061 17996 leader_election.cc:288] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 election: Requested vote from peers 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759), 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737)
I20260619 18:53:02.240717 17639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44127153a83a4bba83d00fb2f6629ffe" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032e4366417c422bb64edde0a27f6ab2"
I20260619 18:53:02.240751 17919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44127153a83a4bba83d00fb2f6629ffe" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0"
I20260619 18:53:02.240979 17639 raft_consensus.cc:3037] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:02.240995 17919 raft_consensus.cc:3037] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:02.243996 17919 raft_consensus.cc:2454] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 1.
I20260619 18:53:02.244165 17639 raft_consensus.cc:2454] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 1.
I20260619 18:53:02.244556 17712 leader_election.cc:302] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e6c1321811408b933914fb7a98c76d, 8f0bb2ae4bea47b59e5190f33a09d5b0; no voters:
I20260619 18:53:02.244966 17996 raft_consensus.cc:2781] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:02.245476 17996 raft_consensus.cc:686] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 1 LEADER]: Becoming Leader. State: Replica: 05e6c1321811408b933914fb7a98c76d, State: Running, Role: LEADER
I20260619 18:53:02.246044 17996 consensus_queue.cc:237] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } }
I20260619 18:53:02.251753 17480 catalog_manager.cc:5495] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d reported cstate change: term changed from 0 to 1, leader changed from <none> to 05e6c1321811408b933914fb7a98c76d (127.15.91.2). New cstate: current_term: 1 leader_uuid: "05e6c1321811408b933914fb7a98c76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } health_report { overall_health: UNKNOWN } } }
W20260619 18:53:02.300891 17690 tablet.cc:2369] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 18:53:02 dist-test-slave-dm3f krb5kdc[17437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895179, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:02.332540 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.317529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.15.91.62:33063)
0619 18:53:02.317722 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:02.317727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:02.318402 (+ 675us) server_negotiation.cc:407] Connection header received
0619 18:53:02.318556 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:02.318559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:02.318624 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:02.318774 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:02.320086 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.320896 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.322261 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.322878 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.323035 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.326631 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:02.326644 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:02.326648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:02.326679 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:02.329224 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:02.330176 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:02.330183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:02.330189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:02.330368 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:02.330851 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:02.330856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:02.330860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:02.331187 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:02.331364 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:02.331806 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 18:53:02.332206 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":197,"server-negotiator.queue_time_us":71}
I20260619 18:53:02.332584 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.317550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:33063 (local address 127.0.0.1:36888)
0619 18:53:02.318270 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:02.318292 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:02.318447 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:02.318819 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:02.318825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:02.319590 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:02.319887 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.319898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.321074 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.321079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.322085 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.322100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.323091 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.323097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.323272 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.324022 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:02.324045 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:02.326379 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:02.329419 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:02.329440 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:02.329450 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:02.330033 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:02.330503 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:02.330509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:02.330513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:02.330663 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:02.331353 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:02.331363 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:02.331537 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 18:53:02.331720 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 18:53:02.332088 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":1,"thread_start_us":422,"threads_started":1}
I20260619 18:53:02.338383 17480 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:36888:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:53:02.339566 17480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:02.347095 17619 tablet_service.cc:1467] Processing CreateTablet for tablet a6d2d45dcbab4c0c8c43a57914941b68 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e944769478fe4f7c9b9a2672a1edf39c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:53:02.347379 17899 tablet_service.cc:1467] Processing CreateTablet for tablet a6d2d45dcbab4c0c8c43a57914941b68 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e944769478fe4f7c9b9a2672a1edf39c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:53:02.347193 17759 tablet_service.cc:1467] Processing CreateTablet for tablet a6d2d45dcbab4c0c8c43a57914941b68 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e944769478fe4f7c9b9a2672a1edf39c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:53:02.348189 17619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6d2d45dcbab4c0c8c43a57914941b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:02.348201 17759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6d2d45dcbab4c0c8c43a57914941b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:02.348201 17899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6d2d45dcbab4c0c8c43a57914941b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:02.355043 17988 tablet_bootstrap.cc:492] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2: Bootstrap starting.
I20260619 18:53:02.356069 17989 tablet_bootstrap.cc:492] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Bootstrap starting.
I20260619 18:53:02.356798 17990 tablet_bootstrap.cc:492] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: Bootstrap starting.
I20260619 18:53:02.356913 17988 tablet_bootstrap.cc:654] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:02.357868 17989 tablet_bootstrap.cc:654] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:02.358839 17990 tablet_bootstrap.cc:654] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: Neither blocks nor log segments found. Creating new log.
W20260619 18:53:02.367940 17830 tablet.cc:2369] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:53:02.369239 17989 tablet_bootstrap.cc:492] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0: No bootstrap required, opened a new log
I20260619 18:53:02.369498 17989 ts_tablet_manager.cc:1397] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260619 18:53:02.369645 17990 tablet_bootstrap.cc:492] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: No bootstrap required, opened a new log
I20260619 18:53:02.369947 17990 ts_tablet_manager.cc:1397] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260619 18:53:02.370183 17988 tablet_bootstrap.cc:492] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2: No bootstrap required, opened a new log
I20260619 18:53:02.370378 17988 ts_tablet_manager.cc:1397] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260619 18:53:02.370553 17989 raft_consensus.cc:348] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.371004 17989 raft_consensus.cc:374] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:02.371170 17989 raft_consensus.cc:729] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0bb2ae4bea47b59e5190f33a09d5b0, State: Initialized, Role: FOLLOWER
I20260619 18:53:02.371047 17990 raft_consensus.cc:348] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.371309 17990 raft_consensus.cc:374] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:02.371546 17988 raft_consensus.cc:348] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.371806 17988 raft_consensus.cc:374] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:02.371896 17988 raft_consensus.cc:729] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032e4366417c422bb64edde0a27f6ab2, State: Initialized, Role: FOLLOWER
I20260619 18:53:02.371630 17989 consensus_queue.cc:260] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.372140 17988 consensus_queue.cc:260] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.372387 17995 tablet_replica.cc:439] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } }
I20260619 18:53:02.371413 17990 raft_consensus.cc:729] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e6c1321811408b933914fb7a98c76d, State: Initialized, Role: FOLLOWER
I20260619 18:53:02.372753 17988 ts_tablet_manager.cc:1428] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:53:02.372807 17995 tablet_replica.cc:442] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:02.372994 17990 consensus_queue.cc:260] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.373116 17994 tablet_replica.cc:439] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } }
I20260619 18:53:02.373518 17994 tablet_replica.cc:442] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:02.373449 17996 tablet_replica.cc:439] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } }
I20260619 18:53:02.373773 17996 tablet_replica.cc:442] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:02.373667 17989 ts_tablet_manager.cc:1428] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:53:02.374468 17990 ts_tablet_manager.cc:1428] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:53:02.487629 17996 raft_consensus.cc:482] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:02.487946 17996 raft_consensus.cc:504] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.488783 17996 leader_election.cc:288] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737), 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759)
I20260619 18:53:02.489394 17639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d2d45dcbab4c0c8c43a57914941b68" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032e4366417c422bb64edde0a27f6ab2" is_pre_election: true
I20260619 18:53:02.489444 17919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d2d45dcbab4c0c8c43a57914941b68" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" is_pre_election: true
I20260619 18:53:02.489696 17639 raft_consensus.cc:2454] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 0.
I20260619 18:53:02.489707 17919 raft_consensus.cc:2454] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 0.
I20260619 18:53:02.490274 17710 leader_election.cc:302] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 032e4366417c422bb64edde0a27f6ab2, 05e6c1321811408b933914fb7a98c76d, 8f0bb2ae4bea47b59e5190f33a09d5b0; no voters:
I20260619 18:53:02.490695 17996 raft_consensus.cc:2781] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:02.490846 17996 raft_consensus.cc:482] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:02.490991 17996 raft_consensus.cc:3037] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:02.493151 17996 raft_consensus.cc:504] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.493944 17996 leader_election.cc:288] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 election: Requested vote from peers 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737), 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759)
I20260619 18:53:02.494571 17639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d2d45dcbab4c0c8c43a57914941b68" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032e4366417c422bb64edde0a27f6ab2"
I20260619 18:53:02.494571 17919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d2d45dcbab4c0c8c43a57914941b68" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0"
I20260619 18:53:02.494879 17919 raft_consensus.cc:3037] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:02.494915 17639 raft_consensus.cc:3037] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:02.498092 17639 raft_consensus.cc:2454] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 1.
I20260619 18:53:02.498092 17919 raft_consensus.cc:2454] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 1.
I20260619 18:53:02.498812 17712 leader_election.cc:302] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 032e4366417c422bb64edde0a27f6ab2, 05e6c1321811408b933914fb7a98c76d, 8f0bb2ae4bea47b59e5190f33a09d5b0; no voters:
I20260619 18:53:02.499214 17996 raft_consensus.cc:2781] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:02.499382 17996 raft_consensus.cc:686] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [term 1 LEADER]: Becoming Leader. State: Replica: 05e6c1321811408b933914fb7a98c76d, State: Running, Role: LEADER
I20260619 18:53:02.499656 17996 consensus_queue.cc:237] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.501469 17999 tablet_replica.cc:439] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: TxnStatusTablet state changed. Reason: New leader 05e6c1321811408b933914fb7a98c76d. Latest consensus state: current_term: 1 leader_uuid: "05e6c1321811408b933914fb7a98c76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } }
I20260619 18:53:02.501935 17999 tablet_replica.cc:442] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:53:02.502687 18011 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:53:02.503047 18011 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:53:02.503158 17481 catalog_manager.cc:5495] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d reported cstate change: term changed from 0 to 1, leader changed from <none> to 05e6c1321811408b933914fb7a98c76d (127.15.91.2). New cstate: current_term: 1 leader_uuid: "05e6c1321811408b933914fb7a98c76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } health_report { overall_health: UNKNOWN } } }
I20260619 18:53:02.528054 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.520740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.15.91.2:34755)
0619 18:53:02.520940 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:02.520947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:02.521074 (+ 127us) server_negotiation.cc:407] Connection header received
0619 18:53:02.521425 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:02.521430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:02.521516 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:02.521676 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:02.522139 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.523048 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.525296 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.525798 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.525951 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.526668 (+ 717us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:02.527390 (+ 722us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:02.527518 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:02.527716 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 18:53:02.527806 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 18:53:02.528232 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.520655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:34755 (local address 127.0.0.1:50432)
0619 18:53:02.520953 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:02.520976 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:02.521185 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:02.521702 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:02.521707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:02.521720 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:53:02.521920 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.521929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.523260 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.523268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.525027 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.525053 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.526041 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.526048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.526259 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.526368 (+ 109us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:53:02.527533 (+ 1165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:53:02.527565 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:53:02.527675 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 18:53:02.527884 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 18:53:02.538235 17639 raft_consensus.cc:1261] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Refusing update from remote peer 05e6c1321811408b933914fb7a98c76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:02.538270 17919 raft_consensus.cc:1261] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Refusing update from remote peer 05e6c1321811408b933914fb7a98c76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:02.539454 17999 consensus_queue.cc:1035] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:02.539952 17996 consensus_queue.cc:1035] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:02.561394 17994 tablet_replica.cc:439] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2: TxnStatusTablet state changed. Reason: New leader 05e6c1321811408b933914fb7a98c76d. Latest consensus state: current_term: 1 leader_uuid: "05e6c1321811408b933914fb7a98c76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } }
I20260619 18:53:02.561816 17994 tablet_replica.cc:442] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:02.564613 17995 tablet_replica.cc:439] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0: TxnStatusTablet state changed. Reason: New leader 05e6c1321811408b933914fb7a98c76d. Latest consensus state: current_term: 1 leader_uuid: "05e6c1321811408b933914fb7a98c76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } }
I20260619 18:53:02.564929 17995 tablet_replica.cc:442] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:02.566663 18014 mvcc.cc:205] Tried to move back new op lower bound from 7298642667664633856 to 7298642667520983040. Current Snapshot: MvccSnapshot[applied={T|T < 7298642667664633856}]
I20260619 18:53:02.569089 17999 tablet_replica.cc:439] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05e6c1321811408b933914fb7a98c76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } }
I20260619 18:53:02.569011 17996 tablet_replica.cc:439] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05e6c1321811408b933914fb7a98c76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } }
I20260619 18:53:02.569358 17999 tablet_replica.cc:442] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:53:02.569407 18016 mvcc.cc:205] Tried to move back new op lower bound from 7298642667664633856 to 7298642667520983040. Current Snapshot: MvccSnapshot[applied={T|T < 7298642667664633856}]
I20260619 18:53:02.572707 17996 tablet_replica.cc:442] T a6d2d45dcbab4c0c8c43a57914941b68 P 05e6c1321811408b933914fb7a98c76d: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:53:02.573092 18015 mvcc.cc:205] Tried to move back new op lower bound from 7298642667664633856 to 7298642667520983040. Current Snapshot: MvccSnapshot[applied={T|T < 7298642667664633856}]
I20260619 18:53:02.576162 17994 tablet_replica.cc:439] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05e6c1321811408b933914fb7a98c76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } }
I20260619 18:53:02.576476 17994 tablet_replica.cc:442] T a6d2d45dcbab4c0c8c43a57914941b68 P 032e4366417c422bb64edde0a27f6ab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:02.581984 17995 tablet_replica.cc:439] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05e6c1321811408b933914fb7a98c76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } } }
I20260619 18:53:02.582309 17995 tablet_replica.cc:442] T a6d2d45dcbab4c0c8c43a57914941b68 P 8f0bb2ae4bea47b59e5190f33a09d5b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:02.590329 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.578870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.15.91.2:34755)
0619 18:53:02.579052 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:02.579058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:02.584508 (+ 5450us) server_negotiation.cc:407] Connection header received
0619 18:53:02.584745 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:02.584750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:02.584843 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:02.584944 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:02.585841 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.586692 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.588386 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.588858 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.589031 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.589488 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:02.589633 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:02.589746 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:02.589889 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 18:53:02.589983 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:53:02.610842 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.604526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:35147 (local address 127.15.91.2:34755)
0619 18:53:02.604723 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:02.604727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:02.605279 (+ 552us) server_negotiation.cc:407] Connection header received
0619 18:53:02.605576 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:02.605580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:02.605643 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:02.605753 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:02.606271 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.607149 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.608878 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.609339 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.609494 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.609973 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:02.610133 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:02.610252 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:02.610392 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 18:53:02.610462 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:53:02.611126 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:02.604595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:34755 (local address 127.15.91.2:35147)
0619 18:53:02.605157 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:02.605175 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:02.605374 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:02.605763 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:02.605767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:02.605778 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:53:02.606115 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.606125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.607292 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.607301 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.608710 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:02.608728 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:02.609562 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:02.609570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:02.609687 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:02.609746 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:53:02.610248 (+ 502us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:53:02.610258 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:53:02.610365 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 18:53:02.610531 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":280,"threads_started":1}
I20260619 18:53:02.622470 17918 raft_consensus.cc:1261] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Refusing update from remote peer 05e6c1321811408b933914fb7a98c76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:02.622624 17639 raft_consensus.cc:1261] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Refusing update from remote peer 05e6c1321811408b933914fb7a98c76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:02.623162 18019 consensus_queue.cc:1035] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:02.623567 17996 consensus_queue.cc:1035] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:02.714115 17482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:02.714438 17482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:02.718825 17482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e: Sending DeleteTablet for 3 replicas of tablet 44127153a83a4bba83d00fb2f6629ffe
I20260619 18:53:02.720389 17899 tablet_service.cc:1514] Processing DeleteTablet for tablet 44127153a83a4bba83d00fb2f6629ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:02 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:56340
I20260619 18:53:02.720395 17619 tablet_service.cc:1514] Processing DeleteTablet for tablet 44127153a83a4bba83d00fb2f6629ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:02 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:51060
I20260619 18:53:02.722044 18031 tablet_replica.cc:330] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: stopping tablet replica
I20260619 18:53:02.722849 18032 tablet_replica.cc:330] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: stopping tablet replica
I20260619 18:53:02.723264 17759 tablet_service.cc:1514] Processing DeleteTablet for tablet 44127153a83a4bba83d00fb2f6629ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:02 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:50430
I20260619 18:53:02.723925 18032 raft_consensus.cc:2229] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:02.723632 17482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:53:02.723424 18031 raft_consensus.cc:2229] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 18:53:02.724666 17482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:02.725131 18031 raft_consensus.cc:2258] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:02.725368 18032 raft_consensus.cc:2258] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:02.727970 18032 ts_tablet_manager.cc:1905] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:02.732652 18031 ts_tablet_manager.cc:1905] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:02.734109 18033 tablet_replica.cc:330] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: stopping tablet replica
I20260619 18:53:02.734617 18033 raft_consensus.cc:2229] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:02.735314 18033 raft_consensus.cc:2258] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:02.736783 17899 tablet_service.cc:1467] Processing CreateTablet for tablet 70afaa1cb54748f5906d12871fddf546 (DEFAULT_TABLE table=test-table [id=562a16cfcb8247dba99cc840fec4773e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:02.737339 17899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70afaa1cb54748f5906d12871fddf546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:02.737663 18033 ts_tablet_manager.cc:1905] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:02.743036 18032 ts_tablet_manager.cc:1918] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:02.745088 17759 tablet_service.cc:1467] Processing CreateTablet for tablet 70afaa1cb54748f5906d12871fddf546 (DEFAULT_TABLE table=test-table [id=562a16cfcb8247dba99cc840fec4773e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:02.745707 17759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70afaa1cb54748f5906d12871fddf546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:02.747502 18033 ts_tablet_manager.cc:1918] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:02.747735 18033 log.cc:1196] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/44127153a83a4bba83d00fb2f6629ffe
I20260619 18:53:02.748435 18033 ts_tablet_manager.cc:1939] T 44127153a83a4bba83d00fb2f6629ffe P 05e6c1321811408b933914fb7a98c76d: Deleting consensus metadata
I20260619 18:53:02.748925 18031 ts_tablet_manager.cc:1918] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:02.743547 17619 tablet_service.cc:1467] Processing CreateTablet for tablet 70afaa1cb54748f5906d12871fddf546 (DEFAULT_TABLE table=test-table [id=562a16cfcb8247dba99cc840fec4773e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:02.749552 17619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70afaa1cb54748f5906d12871fddf546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:02.751003 17468 catalog_manager.cc:4841] TS 05e6c1321811408b933914fb7a98c76d (127.15.91.2:34755): tablet 44127153a83a4bba83d00fb2f6629ffe (table test-table [id=a23eba9e473c4931a00d66dc95101de3]) successfully deleted
I20260619 18:53:02.756403 18032 log.cc:1196] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/44127153a83a4bba83d00fb2f6629ffe
I20260619 18:53:02.756696 17990 tablet_bootstrap.cc:492] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d: Bootstrap starting.
I20260619 18:53:02.755797 17989 tablet_bootstrap.cc:492] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Bootstrap starting.
I20260619 18:53:02.757071 17988 tablet_bootstrap.cc:492] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2: Bootstrap starting.
I20260619 18:53:02.757561 18032 ts_tablet_manager.cc:1939] T 44127153a83a4bba83d00fb2f6629ffe P 032e4366417c422bb64edde0a27f6ab2: Deleting consensus metadata
I20260619 18:53:02.758605 17990 tablet_bootstrap.cc:654] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:02.758925 17989 tablet_bootstrap.cc:654] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:02.759150 17988 tablet_bootstrap.cc:654] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:02.759896 17467 catalog_manager.cc:4841] TS 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737): tablet 44127153a83a4bba83d00fb2f6629ffe (table test-table [id=a23eba9e473c4931a00d66dc95101de3]) successfully deleted
I20260619 18:53:02.760401 18031 log.cc:1196] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/44127153a83a4bba83d00fb2f6629ffe
I20260619 18:53:02.761579 18031 ts_tablet_manager.cc:1939] T 44127153a83a4bba83d00fb2f6629ffe P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting consensus metadata
I20260619 18:53:02.764701 17469 catalog_manager.cc:4841] TS 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759): tablet 44127153a83a4bba83d00fb2f6629ffe (table test-table [id=a23eba9e473c4931a00d66dc95101de3]) successfully deleted
I20260619 18:53:02.768769 17988 tablet_bootstrap.cc:492] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2: No bootstrap required, opened a new log
I20260619 18:53:02.769097 17988 ts_tablet_manager.cc:1397] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260619 18:53:02.769320 17989 tablet_bootstrap.cc:492] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0: No bootstrap required, opened a new log
I20260619 18:53:02.769518 17989 ts_tablet_manager.cc:1397] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260619 18:53:02.769553 17990 tablet_bootstrap.cc:492] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d: No bootstrap required, opened a new log
I20260619 18:53:02.769766 17990 ts_tablet_manager.cc:1397] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260619 18:53:02.769994 17988 raft_consensus.cc:348] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.770259 17988 raft_consensus.cc:374] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:02.770346 17988 raft_consensus.cc:729] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032e4366417c422bb64edde0a27f6ab2, State: Initialized, Role: FOLLOWER
I20260619 18:53:02.770380 17989 raft_consensus.cc:348] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.770653 17989 raft_consensus.cc:374] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:02.770757 17989 raft_consensus.cc:729] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0bb2ae4bea47b59e5190f33a09d5b0, State: Initialized, Role: FOLLOWER
I20260619 18:53:02.770605 17990 raft_consensus.cc:348] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.770632 17988 consensus_queue.cc:260] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.770898 17990 raft_consensus.cc:374] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:02.770994 17990 raft_consensus.cc:729] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e6c1321811408b933914fb7a98c76d, State: Initialized, Role: FOLLOWER
I20260619 18:53:02.771070 17989 consensus_queue.cc:260] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.771265 17990 consensus_queue.cc:260] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.771888 17990 ts_tablet_manager.cc:1428] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:53:02.772276 17989 ts_tablet_manager.cc:1428] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 18:53:02.773773 17988 ts_tablet_manager.cc:1428] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 18:53:02.957093 17996 raft_consensus.cc:482] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:02.957340 17996 raft_consensus.cc:504] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.958189 17996 leader_election.cc:288] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737), 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759)
I20260619 18:53:02.958931 17638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70afaa1cb54748f5906d12871fddf546" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032e4366417c422bb64edde0a27f6ab2" is_pre_election: true
I20260619 18:53:02.959079 17918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70afaa1cb54748f5906d12871fddf546" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" is_pre_election: true
I20260619 18:53:02.959203 17638 raft_consensus.cc:2454] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 0.
I20260619 18:53:02.959371 17918 raft_consensus.cc:2454] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 0.
I20260619 18:53:02.959936 17712 leader_election.cc:302] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 032e4366417c422bb64edde0a27f6ab2, 05e6c1321811408b933914fb7a98c76d, 8f0bb2ae4bea47b59e5190f33a09d5b0; no voters:
I20260619 18:53:02.960325 17996 raft_consensus.cc:2781] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:02.960506 17996 raft_consensus.cc:482] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:02.960606 17996 raft_consensus.cc:3037] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:02.964103 17996 raft_consensus.cc:504] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.964915 17996 leader_election.cc:288] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 election: Requested vote from peers 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737), 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759)
I20260619 18:53:02.965626 17638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70afaa1cb54748f5906d12871fddf546" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032e4366417c422bb64edde0a27f6ab2"
I20260619 18:53:02.965656 17918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70afaa1cb54748f5906d12871fddf546" candidate_uuid: "05e6c1321811408b933914fb7a98c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0"
I20260619 18:53:02.965869 17638 raft_consensus.cc:3037] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:02.965935 17918 raft_consensus.cc:3037] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:02.969200 17638 raft_consensus.cc:2454] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 1.
I20260619 18:53:02.969296 17918 raft_consensus.cc:2454] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e6c1321811408b933914fb7a98c76d in term 1.
I20260619 18:53:02.970081 17712 leader_election.cc:302] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e6c1321811408b933914fb7a98c76d, 8f0bb2ae4bea47b59e5190f33a09d5b0; no voters:
I20260619 18:53:02.970669 17996 raft_consensus.cc:2781] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:02.970858 17996 raft_consensus.cc:686] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 1 LEADER]: Becoming Leader. State: Replica: 05e6c1321811408b933914fb7a98c76d, State: Running, Role: LEADER
I20260619 18:53:02.971174 17996 consensus_queue.cc:237] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:02.974579 17480 catalog_manager.cc:5495] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d reported cstate change: term changed from 0 to 1, leader changed from <none> to 05e6c1321811408b933914fb7a98c76d (127.15.91.2). New cstate: current_term: 1 leader_uuid: "05e6c1321811408b933914fb7a98c76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } health_report { overall_health: UNKNOWN } } }
I20260619 18:53:03.021250 17918 raft_consensus.cc:1261] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Refusing update from remote peer 05e6c1321811408b933914fb7a98c76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:03.021847 17638 raft_consensus.cc:1261] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Refusing update from remote peer 05e6c1321811408b933914fb7a98c76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:03.021930 18038 consensus_queue.cc:1035] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:03.022451 17996 consensus_queue.cc:1035] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:53:03.052044 18014 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 18:53:04.687168 17480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:04.687510 17480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:04.692188 17480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e: Sending DeleteTablet for 3 replicas of tablet 70afaa1cb54748f5906d12871fddf546
I20260619 18:53:04.693239 17619 tablet_service.cc:1514] Processing DeleteTablet for tablet 70afaa1cb54748f5906d12871fddf546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:04 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:51060
I20260619 18:53:04.693378 17899 tablet_service.cc:1514] Processing DeleteTablet for tablet 70afaa1cb54748f5906d12871fddf546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:04 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:56340
I20260619 18:53:04.693432 17759 tablet_service.cc:1514] Processing DeleteTablet for tablet 70afaa1cb54748f5906d12871fddf546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:04 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:50430
I20260619 18:53:04.706255 18061 tablet_replica.cc:330] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0: stopping tablet replica
I20260619 18:53:04.706835 18061 raft_consensus.cc:2229] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:04.707319 18061 raft_consensus.cc:2258] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:04.708190 18060 tablet_replica.cc:330] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2: stopping tablet replica
I20260619 18:53:04.708860 18062 tablet_replica.cc:330] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d: stopping tablet replica
I20260619 18:53:04.709131 18062 raft_consensus.cc:2229] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:04.709154 18061 ts_tablet_manager.cc:1905] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:04.709261 18060 raft_consensus.cc:2229] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:04.709632 18060 raft_consensus.cc:2258] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:04.709673 18062 raft_consensus.cc:2258] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:04.710793 18060 ts_tablet_manager.cc:1905] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:04.711264 18062 ts_tablet_manager.cc:1905] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:04.716928 18061 ts_tablet_manager.cc:1918] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:53:04.717255 18061 log.cc:1196] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/70afaa1cb54748f5906d12871fddf546
I20260619 18:53:04.717787 18061 ts_tablet_manager.cc:1939] T 70afaa1cb54748f5906d12871fddf546 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting consensus metadata
I20260619 18:53:04.719295 18060 ts_tablet_manager.cc:1918] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:53:04.719331 17469 catalog_manager.cc:4841] TS 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759): tablet 70afaa1cb54748f5906d12871fddf546 (table test-table [id=562a16cfcb8247dba99cc840fec4773e]) successfully deleted
I20260619 18:53:04.719601 18060 log.cc:1196] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/70afaa1cb54748f5906d12871fddf546
I20260619 18:53:04.720042 18062 ts_tablet_manager.cc:1918] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:53:04.720101 18060 ts_tablet_manager.cc:1939] T 70afaa1cb54748f5906d12871fddf546 P 032e4366417c422bb64edde0a27f6ab2: Deleting consensus metadata
I20260619 18:53:04.720323 18062 log.cc:1196] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/70afaa1cb54748f5906d12871fddf546
I20260619 18:53:04.720887 18062 ts_tablet_manager.cc:1939] T 70afaa1cb54748f5906d12871fddf546 P 05e6c1321811408b933914fb7a98c76d: Deleting consensus metadata
I20260619 18:53:04.721406 17467 catalog_manager.cc:4841] TS 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737): tablet 70afaa1cb54748f5906d12871fddf546 (table test-table [id=562a16cfcb8247dba99cc840fec4773e]) successfully deleted
I20260619 18:53:04.722373 17468 catalog_manager.cc:4841] TS 05e6c1321811408b933914fb7a98c76d (127.15.91.2:34755): tablet 70afaa1cb54748f5906d12871fddf546 (table test-table [id=562a16cfcb8247dba99cc840fec4773e]) successfully deleted
Jun 19 18:53:04 dist-test-slave-dm3f krb5kdc[17437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:53:04 dist-test-slave-dm3f krb5kdc[17437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:04.758889 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:04.742459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.15.91.62:33063)
0619 18:53:04.743108 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:04.743116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:04.743246 (+ 130us) server_negotiation.cc:407] Connection header received
0619 18:53:04.743524 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:04.743528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:04.743597 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:04.743729 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:04.745184 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.746102 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.747664 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.748107 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.748290 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:04.752603 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:04.752609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:04.752613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:04.752657 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:04.755748 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:04.756676 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:04.756681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:04.756683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:04.756829 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:04.757258 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:04.757262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:04.757264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:04.757582 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:04.757847 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:04.758194 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 18:53:04.758475 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":329,"threads_started":1}
I20260619 18:53:04.763103 17480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:53:04.763800 17480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:04.771451 17759 tablet_service.cc:1467] Processing CreateTablet for tablet 83a429dba37c46fbac0db674fe53766a (DEFAULT_TABLE table=test-table [id=1ba1b26f818d446fbd51a9d2a5855cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:04.771417 17619 tablet_service.cc:1467] Processing CreateTablet for tablet 83a429dba37c46fbac0db674fe53766a (DEFAULT_TABLE table=test-table [id=1ba1b26f818d446fbd51a9d2a5855cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:04.771716 17899 tablet_service.cc:1467] Processing CreateTablet for tablet 83a429dba37c46fbac0db674fe53766a (DEFAULT_TABLE table=test-table [id=1ba1b26f818d446fbd51a9d2a5855cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:04.772140 17899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a429dba37c46fbac0db674fe53766a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:04.772162 17759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a429dba37c46fbac0db674fe53766a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:04.772176 17619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a429dba37c46fbac0db674fe53766a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:04.779553 18071 tablet_bootstrap.cc:492] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0: Bootstrap starting.
I20260619 18:53:04.781574 18071 tablet_bootstrap.cc:654] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:04.786801 18072 tablet_bootstrap.cc:492] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2: Bootstrap starting.
I20260619 18:53:04.787618 18073 tablet_bootstrap.cc:492] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d: Bootstrap starting.
I20260619 18:53:04.789865 18073 tablet_bootstrap.cc:654] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:04.789875 18072 tablet_bootstrap.cc:654] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:04.791065 18071 tablet_bootstrap.cc:492] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0: No bootstrap required, opened a new log
I20260619 18:53:04.791364 18071 ts_tablet_manager.cc:1397] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260619 18:53:04.792629 18072 tablet_bootstrap.cc:492] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2: No bootstrap required, opened a new log
I20260619 18:53:04.792622 18073 tablet_bootstrap.cc:492] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d: No bootstrap required, opened a new log
I20260619 18:53:04.792806 18073 ts_tablet_manager.cc:1397] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:53:04.792840 18072 ts_tablet_manager.cc:1397] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 18:53:04.793723 18073 raft_consensus.cc:348] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:04.793732 18072 raft_consensus.cc:348] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:04.793993 18073 raft_consensus.cc:374] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:04.794018 18072 raft_consensus.cc:374] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:04.794088 18073 raft_consensus.cc:729] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e6c1321811408b933914fb7a98c76d, State: Initialized, Role: FOLLOWER
I20260619 18:53:04.794117 18072 raft_consensus.cc:729] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032e4366417c422bb64edde0a27f6ab2, State: Initialized, Role: FOLLOWER
I20260619 18:53:04.794010 18071 raft_consensus.cc:348] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:04.794293 18071 raft_consensus.cc:374] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:04.794382 18071 raft_consensus.cc:729] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0bb2ae4bea47b59e5190f33a09d5b0, State: Initialized, Role: FOLLOWER
I20260619 18:53:04.794371 18072 consensus_queue.cc:260] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:04.794354 18073 consensus_queue.cc:260] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:04.794654 18071 consensus_queue.cc:260] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:04.795135 18073 ts_tablet_manager.cc:1428] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:53:04.803388 18071 ts_tablet_manager.cc:1428] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0: Time spent starting tablet: real 0.012s user 0.004s sys 0.005s
I20260619 18:53:04.803670 18072 ts_tablet_manager.cc:1428] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2: Time spent starting tablet: real 0.011s user 0.001s sys 0.010s
I20260619 18:53:04.903640 18077 raft_consensus.cc:482] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:04.903895 18077 raft_consensus.cc:504] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:04.905809 18077 leader_election.cc:288] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05e6c1321811408b933914fb7a98c76d (127.15.91.2:34755), 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759)
I20260619 18:53:04.929733 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:04.906468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:32989 (local address 127.15.91.3:37759)
0619 18:53:04.919283 (+ 12815us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:04.919291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:04.920126 (+ 835us) server_negotiation.cc:407] Connection header received
0619 18:53:04.920377 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:04.920383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:04.920476 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:04.920617 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:04.921632 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.922967 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.926674 (+ 3707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.928099 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.928300 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:04.928581 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:04.929159 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 18:53:04.929333 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12641,"thread_start_us":12461,"threads_started":1}
I20260619 18:53:04.929728 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:04.906150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:48003 (local address 127.15.91.2:34755)
0619 18:53:04.919478 (+ 13328us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:04.919486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:04.920126 (+ 640us) server_negotiation.cc:407] Connection header received
0619 18:53:04.920377 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:04.920398 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:04.920476 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:04.920617 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:04.921632 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.922967 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.926674 (+ 3707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.928099 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.928300 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:04.928581 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:04.929160 (+ 579us) server_negotiation.cc:299] Negotiation successful
0619 18:53:04.929340 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13170,"thread_start_us":13160,"threads_started":1}
I20260619 18:53:04.930088 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:04.906148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:37759 (local address 127.15.91.1:32989)
0619 18:53:04.919976 (+ 13828us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:04.919992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:04.920206 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:04.920667 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:04.920672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:04.920708 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:04.921300 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:04.921323 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.923244 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.923252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:04.926334 (+ 3082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:04.926367 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.928489 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.928498 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:04.928844 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:04.928930 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 18:53:04.929062 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 18:53:04.929292 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13713,"mutex_wait_us":400,"spinlock_wait_cycles":21888,"thread_start_us":13466,"threads_started":1}
I20260619 18:53:04.930199 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:04.906092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:34755 (local address 127.15.91.1:48003)
0619 18:53:04.919943 (+ 13851us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:04.919972 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:04.920206 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:04.920667 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:04.920672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:04.920710 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:04.921359 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:04.921373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.923244 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.923252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:04.926334 (+ 3082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:04.926366 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.928489 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.928498 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:04.928843 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:04.928930 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 18:53:04.929062 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 18:53:04.929292 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13668,"mutex_wait_us":424,"spinlock_wait_cycles":44416,"thread_start_us":13576,"threads_started":1}
I20260619 18:53:04.931141 17919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a429dba37c46fbac0db674fe53766a" candidate_uuid: "032e4366417c422bb64edde0a27f6ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" is_pre_election: true
I20260619 18:53:04.931329 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a429dba37c46fbac0db674fe53766a" candidate_uuid: "032e4366417c422bb64edde0a27f6ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e6c1321811408b933914fb7a98c76d" is_pre_election: true
I20260619 18:53:04.931555 17919 raft_consensus.cc:2454] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032e4366417c422bb64edde0a27f6ab2 in term 0.
I20260619 18:53:04.931833 17779 raft_consensus.cc:2454] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032e4366417c422bb64edde0a27f6ab2 in term 0.
I20260619 18:53:04.932245 17572 leader_election.cc:302] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032e4366417c422bb64edde0a27f6ab2, 8f0bb2ae4bea47b59e5190f33a09d5b0; no voters:
I20260619 18:53:04.932722 18077 raft_consensus.cc:2781] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:04.932900 18077 raft_consensus.cc:482] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:04.933024 18077 raft_consensus.cc:3037] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:04.935649 18077 raft_consensus.cc:504] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:04.936682 18077 leader_election.cc:288] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [CANDIDATE]: Term 1 election: Requested vote from peers 05e6c1321811408b933914fb7a98c76d (127.15.91.2:34755), 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759)
I20260619 18:53:04.937256 17919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a429dba37c46fbac0db674fe53766a" candidate_uuid: "032e4366417c422bb64edde0a27f6ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0"
I20260619 18:53:04.937266 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a429dba37c46fbac0db674fe53766a" candidate_uuid: "032e4366417c422bb64edde0a27f6ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e6c1321811408b933914fb7a98c76d"
I20260619 18:53:04.937588 17919 raft_consensus.cc:3037] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:04.937599 17779 raft_consensus.cc:3037] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:04.940238 17919 raft_consensus.cc:2454] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032e4366417c422bb64edde0a27f6ab2 in term 1.
I20260619 18:53:04.940248 17779 raft_consensus.cc:2454] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032e4366417c422bb64edde0a27f6ab2 in term 1.
I20260619 18:53:04.940810 17572 leader_election.cc:302] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032e4366417c422bb64edde0a27f6ab2, 8f0bb2ae4bea47b59e5190f33a09d5b0; no voters:
I20260619 18:53:04.941217 18077 raft_consensus.cc:2781] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:04.941715 18077 raft_consensus.cc:686] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 1 LEADER]: Becoming Leader. State: Replica: 032e4366417c422bb64edde0a27f6ab2, State: Running, Role: LEADER
I20260619 18:53:04.942135 18077 consensus_queue.cc:237] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:04.946235 17480 catalog_manager.cc:5495] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1). New cstate: current_term: 1 leader_uuid: "032e4366417c422bb64edde0a27f6ab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } health_report { overall_health: UNKNOWN } } }
I20260619 18:53:04.996192 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:04.989866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.15.91.1:45737)
0619 18:53:04.990427 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:04.990432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:04.990451 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:53:04.990544 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:04.990547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:04.990626 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:04.990819 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:04.991247 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.992060 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.993471 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:04.993925 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:04.994041 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:04.994633 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:04.995373 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:04.995523 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:04.995702 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 18:53:04.995780 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":308,"threads_started":1}
I20260619 18:53:05.015106 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.008556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:56009 (local address 127.15.91.2:34755)
0619 18:53:05.008810 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:05.008819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:05.009440 (+ 621us) server_negotiation.cc:407] Connection header received
0619 18:53:05.009689 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:05.009694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:05.009766 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:05.009872 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:05.010421 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.011244 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.013148 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.013594 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.013754 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.014267 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:05.014396 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:05.014506 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:05.014653 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:53:05.014781 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 18:53:05.015254 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.008537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:34755 (local address 127.15.91.1:56009)
0619 18:53:05.009284 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:05.009302 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:05.009532 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:05.009889 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:05.009894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:05.009915 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:53:05.010237 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.010246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.011407 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.011413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.012945 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.012966 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.013827 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.013836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.013985 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.014056 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:53:05.014509 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:53:05.014524 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 18:53:05.014629 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 18:53:05.014823 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":386,"threads_started":1}
I20260619 18:53:05.026798 17919 raft_consensus.cc:1261] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Refusing update from remote peer 032e4366417c422bb64edde0a27f6ab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:05.026793 17779 raft_consensus.cc:1261] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Refusing update from remote peer 032e4366417c422bb64edde0a27f6ab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:05.027747 18077 consensus_queue.cc:1035] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:05.028188 18083 consensus_queue.cc:1035] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:05.064234 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.057877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:45737 (local address 127.15.91.2:60019)
0619 18:53:05.058552 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:05.058573 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:05.058701 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:05.059083 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:05.059086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:05.059097 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:53:05.059386 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.059395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.060583 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.060590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.062266 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.062285 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.063190 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.063195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.063331 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.063375 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:53:05.063744 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:53:05.063752 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:53:05.063815 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 18:53:05.063934 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":326,"threads_started":1}
I20260619 18:53:05.064218 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.058454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:60019 (local address 127.15.91.1:45737)
0619 18:53:05.058652 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:05.058659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:05.058680 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:53:05.058879 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:05.058883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:05.058954 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:05.059045 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:05.059551 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.060419 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.062475 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.063007 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.063120 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.063541 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:05.063655 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:05.063724 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:05.063853 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 18:53:05.063923 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:53:05.103430 17481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:05.103760 17481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:05.107383 17481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e: Sending DeleteTablet for 3 replicas of tablet 83a429dba37c46fbac0db674fe53766a
I20260619 18:53:05.108500 17899 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a429dba37c46fbac0db674fe53766a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:05 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:56340
I20260619 18:53:05.108470 17619 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a429dba37c46fbac0db674fe53766a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:05 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:51060
I20260619 18:53:05.108541 17759 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a429dba37c46fbac0db674fe53766a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:05 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:50430
I20260619 18:53:05.108816 18060 tablet_replica.cc:330] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2: stopping tablet replica
I20260619 18:53:05.108817 18061 tablet_replica.cc:330] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0: stopping tablet replica
I20260619 18:53:05.108843 18062 tablet_replica.cc:330] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d: stopping tablet replica
I20260619 18:53:05.109035 18060 raft_consensus.cc:2229] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:05.109035 18061 raft_consensus.cc:2229] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:05.109055 18062 raft_consensus.cc:2229] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:05.109269 18062 raft_consensus.cc:2258] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:05.109251 18061 raft_consensus.cc:2258] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:05.109632 18060 raft_consensus.cc:2258] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:05.110637 18062 ts_tablet_manager.cc:1905] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:05.110649 18061 ts_tablet_manager.cc:1905] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:05.110829 18060 ts_tablet_manager.cc:1905] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:05.118156 18061 ts_tablet_manager.cc:1918] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:05.118011 17481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:53:05.118443 18062 ts_tablet_manager.cc:1918] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:05.118659 18061 log.cc:1196] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/83a429dba37c46fbac0db674fe53766a
I20260619 18:53:05.118765 18062 log.cc:1196] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/83a429dba37c46fbac0db674fe53766a
W20260619 18:53:05.118848 17481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:05.119266 18061 ts_tablet_manager.cc:1939] T 83a429dba37c46fbac0db674fe53766a P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting consensus metadata
I20260619 18:53:05.119402 18062 ts_tablet_manager.cc:1939] T 83a429dba37c46fbac0db674fe53766a P 05e6c1321811408b933914fb7a98c76d: Deleting consensus metadata
I20260619 18:53:05.120671 17469 catalog_manager.cc:4841] TS 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759): tablet 83a429dba37c46fbac0db674fe53766a (table test-table [id=1ba1b26f818d446fbd51a9d2a5855cd5]) successfully deleted
I20260619 18:53:05.120740 17468 catalog_manager.cc:4841] TS 05e6c1321811408b933914fb7a98c76d (127.15.91.2:34755): tablet 83a429dba37c46fbac0db674fe53766a (table test-table [id=1ba1b26f818d446fbd51a9d2a5855cd5]) successfully deleted
I20260619 18:53:05.125069 18060 ts_tablet_manager.cc:1918] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:05.125391 18060 log.cc:1196] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/83a429dba37c46fbac0db674fe53766a
I20260619 18:53:05.125938 18060 ts_tablet_manager.cc:1939] T 83a429dba37c46fbac0db674fe53766a P 032e4366417c422bb64edde0a27f6ab2: Deleting consensus metadata
I20260619 18:53:05.127269 17467 catalog_manager.cc:4841] TS 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737): tablet 83a429dba37c46fbac0db674fe53766a (table test-table [id=1ba1b26f818d446fbd51a9d2a5855cd5]) successfully deleted
I20260619 18:53:05.128276 17899 tablet_service.cc:1467] Processing CreateTablet for tablet ade02608c97e490d9a91356e3154b9e1 (DEFAULT_TABLE table=test-table [id=7495775e474c48b996471d601286435c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:05.128284 17759 tablet_service.cc:1467] Processing CreateTablet for tablet ade02608c97e490d9a91356e3154b9e1 (DEFAULT_TABLE table=test-table [id=7495775e474c48b996471d601286435c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:05.128764 17899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade02608c97e490d9a91356e3154b9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:05.128772 17759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade02608c97e490d9a91356e3154b9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:05.134516 18073 tablet_bootstrap.cc:492] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d: Bootstrap starting.
I20260619 18:53:05.135385 17619 tablet_service.cc:1467] Processing CreateTablet for tablet ade02608c97e490d9a91356e3154b9e1 (DEFAULT_TABLE table=test-table [id=7495775e474c48b996471d601286435c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:05.135905 17619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade02608c97e490d9a91356e3154b9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:05.136379 18073 tablet_bootstrap.cc:654] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:05.140061 18071 tablet_bootstrap.cc:492] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Bootstrap starting.
I20260619 18:53:05.140151 18073 tablet_bootstrap.cc:492] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d: No bootstrap required, opened a new log
I20260619 18:53:05.140312 18073 ts_tablet_manager.cc:1397] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 18:53:05.141228 18073 raft_consensus.cc:348] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:05.141496 18073 raft_consensus.cc:374] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:05.141578 18073 raft_consensus.cc:729] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e6c1321811408b933914fb7a98c76d, State: Initialized, Role: FOLLOWER
I20260619 18:53:05.141986 18071 tablet_bootstrap.cc:654] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:05.141901 18073 consensus_queue.cc:260] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:05.144446 18072 tablet_bootstrap.cc:492] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2: Bootstrap starting.
I20260619 18:53:05.145773 18073 ts_tablet_manager.cc:1428] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 18:53:05.146313 18072 tablet_bootstrap.cc:654] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:05.146540 18071 tablet_bootstrap.cc:492] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0: No bootstrap required, opened a new log
I20260619 18:53:05.146754 18071 ts_tablet_manager.cc:1397] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 18:53:05.147790 18071 raft_consensus.cc:348] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:05.148274 18071 raft_consensus.cc:374] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:05.148424 18071 raft_consensus.cc:729] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0bb2ae4bea47b59e5190f33a09d5b0, State: Initialized, Role: FOLLOWER
I20260619 18:53:05.148864 18071 consensus_queue.cc:260] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:05.149843 18071 ts_tablet_manager.cc:1428] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 18:53:05.152189 18072 tablet_bootstrap.cc:492] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2: No bootstrap required, opened a new log
I20260619 18:53:05.152442 18072 ts_tablet_manager.cc:1397] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260619 18:53:05.153302 18072 raft_consensus.cc:348] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:05.153577 18072 raft_consensus.cc:374] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:05.153676 18072 raft_consensus.cc:729] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032e4366417c422bb64edde0a27f6ab2, State: Initialized, Role: FOLLOWER
I20260619 18:53:05.153962 18072 consensus_queue.cc:260] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:05.154772 18072 ts_tablet_manager.cc:1428] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260619 18:53:05.239974 18078 raft_consensus.cc:482] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:05.240309 18078 raft_consensus.cc:504] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:05.242198 18078 leader_election.cc:288] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737), 05e6c1321811408b933914fb7a98c76d (127.15.91.2:34755)
I20260619 18:53:05.250983 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.242386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:45737 (local address 127.15.91.3:55301)
0619 18:53:05.243141 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:05.243162 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:05.243397 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:05.243937 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:05.243940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:05.243961 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:05.244272 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.244280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.245805 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.245815 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.248392 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.248412 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.250071 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.250078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.250269 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.250296 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:53:05.250416 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 18:53:05.250580 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":584,"threads_started":1}
I20260619 18:53:05.250969 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.242748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:55301 (local address 127.15.91.1:45737)
0619 18:53:05.242961 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:05.242969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:05.243263 (+ 294us) server_negotiation.cc:407] Connection header received
0619 18:53:05.243611 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:05.243618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:05.243698 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:05.243757 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:05.244452 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.245634 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.248636 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.249817 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.250004 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.250125 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:05.250473 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 18:53:05.250628 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:53:05.252090 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.242613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:34755 (local address 127.15.91.3:56627)
0619 18:53:05.244905 (+ 2292us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:05.244927 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:05.245067 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:05.245466 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:05.245471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:05.245496 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:05.246015 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.246027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.247390 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.247396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.249163 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.249180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.251152 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.251160 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.251358 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.251401 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 18:53:05.251515 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 18:53:05.251666 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2125,"thread_start_us":1954,"threads_started":1}
I20260619 18:53:05.252254 17638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade02608c97e490d9a91356e3154b9e1" candidate_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032e4366417c422bb64edde0a27f6ab2" is_pre_election: true
I20260619 18:53:05.252444 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.243334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:56627 (local address 127.15.91.2:34755)
0619 18:53:05.243540 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:05.243548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:05.245046 (+ 1498us) server_negotiation.cc:407] Connection header received
0619 18:53:05.245230 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:05.245236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:05.245321 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:05.245425 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:05.246209 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.247196 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.249375 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.250683 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.250874 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.250987 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:05.251581 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 18:53:05.252128 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:53:05.252606 17638 raft_consensus.cc:2454] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f0bb2ae4bea47b59e5190f33a09d5b0 in term 0.
I20260619 18:53:05.253010 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade02608c97e490d9a91356e3154b9e1" candidate_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e6c1321811408b933914fb7a98c76d" is_pre_election: true
I20260619 18:53:05.253295 17779 raft_consensus.cc:2454] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f0bb2ae4bea47b59e5190f33a09d5b0 in term 0.
I20260619 18:53:05.253273 17850 leader_election.cc:302] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032e4366417c422bb64edde0a27f6ab2, 8f0bb2ae4bea47b59e5190f33a09d5b0; no voters:
I20260619 18:53:05.253777 18078 raft_consensus.cc:2781] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:05.253949 18078 raft_consensus.cc:482] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:05.254066 18078 raft_consensus.cc:3037] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:05.256541 18078 raft_consensus.cc:504] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:05.257385 18078 leader_election.cc:288] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [CANDIDATE]: Term 1 election: Requested vote from peers 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737), 05e6c1321811408b933914fb7a98c76d (127.15.91.2:34755)
I20260619 18:53:05.258060 17638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade02608c97e490d9a91356e3154b9e1" candidate_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032e4366417c422bb64edde0a27f6ab2"
I20260619 18:53:05.258097 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade02608c97e490d9a91356e3154b9e1" candidate_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e6c1321811408b933914fb7a98c76d"
I20260619 18:53:05.258324 17779 raft_consensus.cc:3037] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:05.258332 17638 raft_consensus.cc:3037] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:05.261634 17779 raft_consensus.cc:2454] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f0bb2ae4bea47b59e5190f33a09d5b0 in term 1.
I20260619 18:53:05.262113 17638 raft_consensus.cc:2454] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f0bb2ae4bea47b59e5190f33a09d5b0 in term 1.
I20260619 18:53:05.262224 17851 leader_election.cc:302] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e6c1321811408b933914fb7a98c76d, 8f0bb2ae4bea47b59e5190f33a09d5b0; no voters:
I20260619 18:53:05.262603 18078 raft_consensus.cc:2781] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:05.263190 18078 raft_consensus.cc:686] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 LEADER]: Becoming Leader. State: Replica: 8f0bb2ae4bea47b59e5190f33a09d5b0, State: Running, Role: LEADER
I20260619 18:53:05.263600 18078 consensus_queue.cc:237] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } }
I20260619 18:53:05.267722 17481 catalog_manager.cc:5495] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3). New cstate: current_term: 1 leader_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f0bb2ae4bea47b59e5190f33a09d5b0" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 37759 } health_report { overall_health: HEALTHY } } }
I20260619 18:53:05.312713 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.306903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.15.91.3:37759)
0619 18:53:05.307190 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:05.307195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:05.307216 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:53:05.307473 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:05.307478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:05.307562 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:05.307751 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:05.308146 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.308890 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.310282 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.310702 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.310912 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.311418 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:05.312032 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:05.312202 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:05.312319 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 18:53:05.312415 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260619 18:53:05.332261 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.325257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:34755 (local address 127.15.91.3:39071)
0619 18:53:05.326253 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:05.326274 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:05.326429 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:05.326851 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:05.326855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:05.326865 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:53:05.327152 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.327161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.328641 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.328645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.329778 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.329794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.330938 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.330945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.331101 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.331172 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:53:05.331668 (+ 496us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:53:05.331681 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 18:53:05.331762 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 18:53:05.331889 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":625,"threads_started":1}
I20260619 18:53:05.332249 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.325530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:39071 (local address 127.15.91.2:34755)
0619 18:53:05.325921 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:05.325928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:05.326376 (+ 448us) server_negotiation.cc:407] Connection header received
0619 18:53:05.326567 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:05.326573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:05.326677 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:05.326816 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:05.327306 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.328470 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.329962 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.330680 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.330943 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.331363 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:05.331514 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:05.331614 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:05.331815 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 18:53:05.331917 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260619 18:53:05.341895 17779 raft_consensus.cc:1261] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Refusing update from remote peer 8f0bb2ae4bea47b59e5190f33a09d5b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:05.342077 17638 raft_consensus.cc:1261] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Refusing update from remote peer 8f0bb2ae4bea47b59e5190f33a09d5b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:05.342697 18102 consensus_queue.cc:1035] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e6c1321811408b933914fb7a98c76d" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:05.343155 18078 consensus_queue.cc:1035] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032e4366417c422bb64edde0a27f6ab2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:05.385923 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.378624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:37759 (local address 127.15.91.2:46905)
0619 18:53:05.378866 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:05.378893 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:05.379017 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:05.379738 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:05.379744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:05.379762 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:53:05.379989 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.380000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.381571 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.381578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.383329 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:05.383352 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.384573 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.384579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:05.384782 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.384853 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:53:05.385326 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:53:05.385338 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 18:53:05.385410 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 18:53:05.385581 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 18:53:05.386801 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:05.379224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:46905 (local address 127.15.91.3:37759)
0619 18:53:05.379402 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:05.379409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:05.379428 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:53:05.379501 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:05.379507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:05.379594 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:05.379701 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:05.380160 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.381375 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.383589 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:05.384298 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:05.384430 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:05.385047 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:05.385203 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:05.385278 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:05.386279 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0619 18:53:05.386375 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 18:53:05.389838 18044 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 18:53:07.015565 17480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:07.015887 17480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:07.021780 17480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6303b1da75ec4564be2c76382ed15b9e: Sending DeleteTablet for 3 replicas of tablet ade02608c97e490d9a91356e3154b9e1
I20260619 18:53:07.022886 17759 tablet_service.cc:1514] Processing DeleteTablet for tablet ade02608c97e490d9a91356e3154b9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:07 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:50430
I20260619 18:53:07.022876 17619 tablet_service.cc:1514] Processing DeleteTablet for tablet ade02608c97e490d9a91356e3154b9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:07 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:51060
I20260619 18:53:07.022874 17899 tablet_service.cc:1514] Processing DeleteTablet for tablet ade02608c97e490d9a91356e3154b9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:07 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:56340
I20260619 18:53:07.025363 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 17553
I20260619 18:53:07.035821 18136 tablet_replica.cc:330] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d: stopping tablet replica
I20260619 18:53:07.036242 18136 raft_consensus.cc:2229] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:07.036679 18136 raft_consensus.cc:2258] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:07.039108 18136 ts_tablet_manager.cc:1905] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:07.039412 18134 tablet_replica.cc:330] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2: stopping tablet replica
I20260619 18:53:07.040316 18134 raft_consensus.cc:2229] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:07.040757 18134 raft_consensus.cc:2258] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:07.042009 18134 ts_tablet_manager.cc:1905] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:07.046222 18135 tablet_replica.cc:330] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0: stopping tablet replica
I20260619 18:53:07.046586 18135 raft_consensus.cc:2229] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:07.047367 18135 raft_consensus.cc:2258] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:07.047480 18136 ts_tablet_manager.cc:1918] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:53:07.048532 18136 log.cc:1196] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/ade02608c97e490d9a91356e3154b9e1
I20260619 18:53:07.048635 18135 ts_tablet_manager.cc:1905] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:07.049242 18136 ts_tablet_manager.cc:1939] T ade02608c97e490d9a91356e3154b9e1 P 05e6c1321811408b933914fb7a98c76d: Deleting consensus metadata
I20260619 18:53:07.049961 18134 ts_tablet_manager.cc:1918] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:53:07.050180 18134 log.cc:1196] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/ade02608c97e490d9a91356e3154b9e1
I20260619 18:53:07.050658 17468 catalog_manager.cc:4841] TS 05e6c1321811408b933914fb7a98c76d (127.15.91.2:34755): tablet ade02608c97e490d9a91356e3154b9e1 (table test-table [id=7495775e474c48b996471d601286435c]) successfully deleted
I20260619 18:53:07.050765 18134 ts_tablet_manager.cc:1939] T ade02608c97e490d9a91356e3154b9e1 P 032e4366417c422bb64edde0a27f6ab2: Deleting consensus metadata
I20260619 18:53:07.052544 17467 catalog_manager.cc:4841] TS 032e4366417c422bb64edde0a27f6ab2 (127.15.91.1:45737): tablet ade02608c97e490d9a91356e3154b9e1 (table test-table [id=7495775e474c48b996471d601286435c]) successfully deleted
I20260619 18:53:07.055280 18135 ts_tablet_manager.cc:1918] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:53:07.055496 18135 log.cc:1196] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/ade02608c97e490d9a91356e3154b9e1
I20260619 18:53:07.055994 18135 ts_tablet_manager.cc:1939] T ade02608c97e490d9a91356e3154b9e1 P 8f0bb2ae4bea47b59e5190f33a09d5b0: Deleting consensus metadata
I20260619 18:53:07.057256 17469 catalog_manager.cc:4841] TS 8f0bb2ae4bea47b59e5190f33a09d5b0 (127.15.91.3:37759): tablet ade02608c97e490d9a91356e3154b9e1 (table test-table [id=7495775e474c48b996471d601286435c]) successfully deleted
Jun 19 18:53:07 dist-test-slave-dm3f krb5kdc[17437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:53:07.066485 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:07.028299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.15.91.1:45737)
0619 18:53:07.039569 (+ 11270us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:07.039580 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:07.040573 (+ 993us) server_negotiation.cc:407] Connection header received
0619 18:53:07.040674 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:07.040683 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:07.040804 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:07.040955 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:07.043648 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:07.045435 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:07.048170 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:07.048918 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:07.049080 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:07.058910 (+ 9830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:07.058929 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:07.058948 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:07.059001 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:07.062627 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:07.063582 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:07.063591 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:07.063593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:07.063774 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:07.064291 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:07.064295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:07.064297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:07.064712 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:07.064991 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:07.065622 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 18:53:07.066080 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11106,"thread_start_us":11220,"threads_started":1}
W20260619 18:53:07.067675 17684 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55482
W20260619 18:53:07.068671 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 17553: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:53:07.068965 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 17553
I20260619 18:53:07.099817 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 17693
Jun 19 18:53:07 dist-test-slave-dm3f krb5kdc[17437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:53:07.119340 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:07.100731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.15.91.2:34755)
0619 18:53:07.101353 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:07.101358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:07.101384 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:53:07.101450 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:07.101454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:07.101530 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:07.101630 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:07.103426 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:07.104221 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:07.106321 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:07.106953 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:07.107094 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:07.112625 (+ 5531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:07.112644 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:07.112661 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:07.112706 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:07.116203 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:07.117140 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:07.117149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:07.117151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:07.117325 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:07.117778 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:07.117782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:07.117785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:07.118188 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:07.118477 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:07.118828 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 18:53:07.119050 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":336,"threads_started":1}
W20260619 18:53:07.120317 17824 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49940
W20260619 18:53:07.121253 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 17693: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:53:07.121440 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 17693
I20260619 18:53:07.148901 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 17833
Jun 19 18:53:07 dist-test-slave-dm3f krb5kdc[17437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:53:07.165570 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:07.149734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.15.91.3:37759)
0619 18:53:07.150258 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:07.150263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:07.150282 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:53:07.150339 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:07.150343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:07.150404 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:07.150533 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:07.151985 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:07.152787 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:07.154141 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:07.154580 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:07.154693 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:07.159658 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:07.159674 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:07.159691 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:07.159737 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:07.162704 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:07.163616 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:07.163624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:07.163626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:07.163782 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:07.164161 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:07.164164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:07.164166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:07.164560 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:07.164811 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:07.165094 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 18:53:07.165285 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":275,"threads_started":1}
W20260619 18:53:07.166376 17964 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39956
W20260619 18:53:07.167140 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 17833: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:53:07.167351 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 17833
I20260619 18:53:07.190403 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 17450
I20260619 18:53:07.191761 17522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:07.313226 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 17450
2026-06-19T18:53:07Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9040 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:53:07 dist-test-slave-dm3f krb5kdc[18144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:53:07 dist-test-slave-dm3f krb5kdc[18144](info): set up 2 sockets
Jun 19 18:53:07 dist-test-slave-dm3f krb5kdc[18144](info): commencing operation
krb5kdc: starting...
W20260619 18:53:08.419309 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:53:08 dist-test-slave-dm3f krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:53:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:53:08.623212 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:42567
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:42567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:08.977264 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:08.977602 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:08.977777 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:08.985488 18157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:08.985627 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:08.985723 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:08.985808 18157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:08.985867 18157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:08.996418 18157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:42567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:42567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:08.997939 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:08.999439 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:09.012729 18162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:09.012773 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:09.012815 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:09.013942 18157 server_base.cc:1034] running on GCE node
I20260619 18:53:09.014947 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:09.016722 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:09.018062 18157 hybrid_clock.cc:648] HybridClock initialized: now 1781895189018010 us; error 74 us; skew 500 ppm
I20260619 18:53:09.018533 18157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:09 dist-test-slave-dm3f krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895189, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:09.022112 18157 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:53:09.024143 18157 webserver.cc:460] Webserver started at http://127.15.91.62:44425/ using document root <none> and password file <none>
I20260619 18:53:09.025012 18157 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:09.025197 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:09.025588 18157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:09.029148 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "8130d9dab069489da1a5b123076dab5e"
format_stamp: "Formatted at 2026-06-19 18:53:09 on dist-test-slave-dm3f"
server_key: "3ffb575814999b4bcbdcd95d46ada56e"
server_key_iv: "747d650f845315da0fa04669240ddc4b"
server_key_version: "encryptionkey@0"
I20260619 18:53:09.030171 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "8130d9dab069489da1a5b123076dab5e"
format_stamp: "Formatted at 2026-06-19 18:53:09 on dist-test-slave-dm3f"
server_key: "3ffb575814999b4bcbdcd95d46ada56e"
server_key_iv: "747d650f845315da0fa04669240ddc4b"
server_key_version: "encryptionkey@0"
I20260619 18:53:09.036964 18157 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:53:09.041378 18172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:09.043145 18157 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:53:09.043406 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "8130d9dab069489da1a5b123076dab5e"
format_stamp: "Formatted at 2026-06-19 18:53:09 on dist-test-slave-dm3f"
server_key: "3ffb575814999b4bcbdcd95d46ada56e"
server_key_iv: "747d650f845315da0fa04669240ddc4b"
server_key_version: "encryptionkey@0"
I20260619 18:53:09.043668 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:09.064932 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:09.069705 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:09.070066 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:09.091629 18157 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:42567
I20260619 18:53:09.091657 18223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:42567 every 8 connection(s)
I20260619 18:53:09.093313 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:09.094134 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 18157
I20260619 18:53:09.094406 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:09.095141 15724 external_mini_cluster.cc:1428] Setting key 15d17d723eb3b161e1f6f3776c878f44
I20260619 18:53:09.100220 18224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:09.117826 18224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e: Bootstrap starting.
Jun 19 18:53:09 dist-test-slave-dm3f krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:09.121651 18224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:09.123059 18224 log.cc:824] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:09.126360 18224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e: No bootstrap required, opened a new log
I20260619 18:53:09.128295 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:09.097755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.15.91.62:42567)
0619 18:53:09.103513 (+ 5758us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:09.103538 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:09.104297 (+ 759us) server_negotiation.cc:407] Connection header received
0619 18:53:09.106259 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:09.106290 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:09.106937 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:09.107370 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:09.109457 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:09.111056 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:09.113155 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:09.114042 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:09.114249 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:09.119711 (+ 5462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:09.119722 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:09.119740 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:09.119803 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:09.123475 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:09.124399 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:09.124407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:09.124415 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:09.124638 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:09.125105 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:09.125112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:09.125121 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:09.125725 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:09.126082 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:09.126588 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 18:53:09.127174 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5406,"thread_start_us":5098,"threads_started":1}
I20260619 18:53:09.135725 18224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130d9dab069489da1a5b123076dab5e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42567 } }
I20260619 18:53:09.136246 18224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:09.136382 18224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8130d9dab069489da1a5b123076dab5e, State: Initialized, Role: FOLLOWER
I20260619 18:53:09.137107 18224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130d9dab069489da1a5b123076dab5e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42567 } }
I20260619 18:53:09.137408 18224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:09.137579 18224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:09.137778 18224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:09.140589 18224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130d9dab069489da1a5b123076dab5e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42567 } }
I20260619 18:53:09.141188 18224 leader_election.cc:302] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8130d9dab069489da1a5b123076dab5e; no voters:
I20260619 18:53:09.141784 18224 leader_election.cc:288] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:09.141966 18229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:09.142753 18229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [term 1 LEADER]: Becoming Leader. State: Replica: 8130d9dab069489da1a5b123076dab5e, State: Running, Role: LEADER
I20260619 18:53:09.143390 18229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130d9dab069489da1a5b123076dab5e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42567 } }
I20260619 18:53:09.144172 18224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:09.146440 18231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8130d9dab069489da1a5b123076dab5e. Latest consensus state: current_term: 1 leader_uuid: "8130d9dab069489da1a5b123076dab5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130d9dab069489da1a5b123076dab5e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42567 } } }
I20260619 18:53:09.147222 18231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:09.147297 18230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8130d9dab069489da1a5b123076dab5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8130d9dab069489da1a5b123076dab5e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42567 } } }
I20260619 18:53:09.147627 18230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:09.157034 18236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:09.163609 18236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:09.174590 18236 catalog_manager.cc:1347] Generated new cluster ID: ce97cbc8514d4052976032b24e2b644a
I20260619 18:53:09.174793 18236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:09.188320 18236 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:09.189836 18236 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:09.196978 18236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8130d9dab069489da1a5b123076dab5e: Generated new TSK 0
I20260619 18:53:09.198150 18236 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:53:09.256891 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42567
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:09.623927 18250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:09.624261 18250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:09.624361 18250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:09.632313 18250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:09.632453 18250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:09.632583 18250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:53:09.643254 18250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:09.644749 18250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:09.646299 18250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:09.662046 18256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:09.662058 18255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:09.662810 18258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:09.668545 18250 server_base.cc:1034] running on GCE node
I20260619 18:53:09.669377 18250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:09.670526 18250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:09.671765 18250 hybrid_clock.cc:648] HybridClock initialized: now 1781895189671705 us; error 99 us; skew 500 ppm
I20260619 18:53:09.672163 18250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:09 dist-test-slave-dm3f krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895189, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:09.677467 18250 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:53:09.683864 18250 webserver.cc:460] Webserver started at http://127.15.91.1:42143/ using document root <none> and password file <none>
I20260619 18:53:09.685092 18250 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:09.685276 18250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:09.685690 18250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:09.690865 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "2b1999524f154542aa78366bb4da55db"
format_stamp: "Formatted at 2026-06-19 18:53:09 on dist-test-slave-dm3f"
server_key: "50612092fa1599b456b7892243cc42b7"
server_key_iv: "00df2ef4ee6ffd8ee7babec7d45e2f90"
server_key_version: "encryptionkey@0"
I20260619 18:53:09.692188 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "2b1999524f154542aa78366bb4da55db"
format_stamp: "Formatted at 2026-06-19 18:53:09 on dist-test-slave-dm3f"
server_key: "50612092fa1599b456b7892243cc42b7"
server_key_iv: "00df2ef4ee6ffd8ee7babec7d45e2f90"
server_key_version: "encryptionkey@0"
I20260619 18:53:09.704049 18250 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.007s sys 0.001s
I20260619 18:53:09.711954 18265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:09.714432 18250 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260619 18:53:09.714743 18250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "2b1999524f154542aa78366bb4da55db"
format_stamp: "Formatted at 2026-06-19 18:53:09 on dist-test-slave-dm3f"
server_key: "50612092fa1599b456b7892243cc42b7"
server_key_iv: "00df2ef4ee6ffd8ee7babec7d45e2f90"
server_key_version: "encryptionkey@0"
I20260619 18:53:09.715116 18250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:09.740025 18250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:09.748042 18250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:09.748587 18250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:09.750622 18250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:09.753005 18250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:09.753141 18250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:09.753293 18250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:09.753368 18250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:09.792199 18250 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:40567
I20260619 18:53:09.792241 18377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:40567 every 8 connection(s)
I20260619 18:53:09.793694 18250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:53:09.799930 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 18250
I20260619 18:53:09.800173 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:53:09.800999 15724 external_mini_cluster.cc:1428] Setting key 7a4b0ab8d03fb39e7c9da30869e6689d
Jun 19 18:53:09 dist-test-slave-dm3f krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895189, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:09.821028 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:09.796646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:40887 (local address 127.15.91.62:42567)
0619 18:53:09.797190 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:09.797195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:09.798217 (+ 1022us) server_negotiation.cc:407] Connection header received
0619 18:53:09.801028 (+ 2811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:09.801033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:09.801161 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:09.801262 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:09.803576 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:09.804312 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:09.807417 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:09.808093 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:09.808228 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:09.812916 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:09.812926 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:09.812935 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:09.812995 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:09.816974 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:09.818064 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:09.818074 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:09.818081 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:09.818277 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:09.818806 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:09.818815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:09.818821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:09.819253 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:09.819430 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:09.820270 (+ 840us) server_negotiation.cc:299] Negotiation successful
0619 18:53:09.820564 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:53:09.821924 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:09.797187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42567 (local address 127.15.91.1:40887)
0619 18:53:09.798012 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:09.798061 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:09.800540 (+ 2479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:09.801626 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:09.801641 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:09.802389 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:09.803300 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:09.803325 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:09.804531 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:09.804541 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:09.807135 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:09.807167 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:09.808430 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:09.808442 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:09.808851 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:09.809720 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:09.809747 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:09.812539 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:09.817262 (+ 4723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:09.817277 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:09.817303 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:09.817860 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:09.818416 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:09.818420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:09.818422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:09.818634 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:09.819409 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:09.819417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:09.819817 (+ 400us) client_negotiation.cc:769] Sending connection context
0619 18:53:09.820077 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 18:53:09.820548 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":322,"threads_started":1}
I20260619 18:53:09.823953 18378 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42567
I20260619 18:53:09.824347 18378 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:53:09.825314 18378 heartbeater.cc:511] Master 127.15.91.62:42567 requested a full tablet report, sending...
I20260619 18:53:09.828090 18189 ts_manager.cc:194] Registered new tserver with Master: 2b1999524f154542aa78366bb4da55db (127.15.91.1:40567)
I20260619 18:53:09.830380 18189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:40887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:53:09.850365 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42567
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:10.207607 18384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:10.207998 18384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:10.208163 18384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:10.215757 18384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:10.215955 18384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:10.216147 18384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:53:10.226706 18384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:10.228245 18384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:10.229718 18384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:10.244840 18389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:10.244905 18390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:10.245291 18392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:10.246075 18384 server_base.cc:1034] running on GCE node
I20260619 18:53:10.247296 18384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:10.248574 18384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:10.249861 18384 hybrid_clock.cc:648] HybridClock initialized: now 1781895190249759 us; error 103 us; skew 500 ppm
I20260619 18:53:10.250298 18384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:10 dist-test-slave-dm3f krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895190, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:10.254021 18384 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:53:10.256263 18384 webserver.cc:460] Webserver started at http://127.15.91.2:38667/ using document root <none> and password file <none>
I20260619 18:53:10.257264 18384 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:10.257447 18384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:10.257809 18384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:10.261279 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "1434e62b6aa64adcb6d55af337ecfd51"
format_stamp: "Formatted at 2026-06-19 18:53:10 on dist-test-slave-dm3f"
server_key: "cc9f40d2d964f27d9f37c1daadef9a5b"
server_key_iv: "b31146d61baf6aea9d22e868df42f428"
server_key_version: "encryptionkey@0"
I20260619 18:53:10.262293 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "1434e62b6aa64adcb6d55af337ecfd51"
format_stamp: "Formatted at 2026-06-19 18:53:10 on dist-test-slave-dm3f"
server_key: "cc9f40d2d964f27d9f37c1daadef9a5b"
server_key_iv: "b31146d61baf6aea9d22e868df42f428"
server_key_version: "encryptionkey@0"
I20260619 18:53:10.269178 18384 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:53:10.273730 18399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:10.275731 18384 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:53:10.275980 18384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "1434e62b6aa64adcb6d55af337ecfd51"
format_stamp: "Formatted at 2026-06-19 18:53:10 on dist-test-slave-dm3f"
server_key: "cc9f40d2d964f27d9f37c1daadef9a5b"
server_key_iv: "b31146d61baf6aea9d22e868df42f428"
server_key_version: "encryptionkey@0"
I20260619 18:53:10.276239 18384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:10.296058 18384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:10.301997 18384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:10.302376 18384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:10.303648 18384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:10.305315 18384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:10.305447 18384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:10.305648 18384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:10.305725 18384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:10.342635 18384 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:37489
I20260619 18:53:10.342674 18511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:37489 every 8 connection(s)
I20260619 18:53:10.344118 18384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:53:10.351492 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 18384
I20260619 18:53:10.351713 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:53:10.352191 15724 external_mini_cluster.cc:1428] Setting key e6b56af8f34ed857b51debf087c5b071
Jun 19 18:53:10 dist-test-slave-dm3f krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895190, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:10.371503 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:10.347037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:55931 (local address 127.15.91.62:42567)
0619 18:53:10.347540 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:10.347545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:10.348737 (+ 1192us) server_negotiation.cc:407] Connection header received
0619 18:53:10.350940 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:10.350946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:10.351057 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:10.351186 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:10.353253 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:10.354367 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:10.357470 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:10.357959 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:10.358093 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:10.363152 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:10.363163 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:10.363172 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:10.363231 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:10.367267 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:10.368298 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:10.368308 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:10.368314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:10.368516 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:10.369044 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:10.369053 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:10.369059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:10.369516 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:10.369706 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:10.370624 (+ 918us) server_negotiation.cc:299] Negotiation successful
0619 18:53:10.370997 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":283,"threads_started":1}
I20260619 18:53:10.372308 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:10.347623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42567 (local address 127.15.91.2:55931)
0619 18:53:10.348541 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:10.348607 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:10.350498 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:10.351497 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:10.351508 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:10.352210 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:10.353043 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:10.353083 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:10.354603 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:10.354610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:10.357168 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:10.357201 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:10.358272 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:10.358283 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:10.358694 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:10.359613 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:10.359653 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:10.362713 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:10.367512 (+ 4799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:10.367525 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:10.367543 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:10.368099 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:10.368652 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:10.368656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:10.368658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:10.368889 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:10.369699 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:10.369706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:10.370138 (+ 432us) client_negotiation.cc:769] Sending connection context
0619 18:53:10.370429 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 18:53:10.370989 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":382,"threads_started":1}
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
I20260619 18:53:10.374210 18512 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42567
I20260619 18:53:10.374558 18512 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:10.375484 18512 heartbeater.cc:511] Master 127.15.91.62:42567 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
I20260619 18:53:10.377393 18189 ts_manager.cc:194] Registered new tserver with Master: 1434e62b6aa64adcb6d55af337ecfd51 (127.15.91.2:37489)
I20260619 18:53:10.378309 18189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:55931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:53:10.402287 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42567
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:10.756716 18518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:10.757037 18518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:10.757223 18518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:10.764927 18518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:10.765075 18518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:10.765275 18518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:53:10.775951 18518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:10.777424 18518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:10.778918 18518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:10.797564 18526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:10.798043 18524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:10.798306 18523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:10.800843 18518 server_base.cc:1034] running on GCE node
I20260619 18:53:10.801692 18518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:10.802800 18518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:10.804100 18518 hybrid_clock.cc:648] HybridClock initialized: now 1781895190804046 us; error 59 us; skew 500 ppm
I20260619 18:53:10.804607 18518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:10 dist-test-slave-dm3f krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895190, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:10.808260 18518 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:53:10.810063 18518 webserver.cc:460] Webserver started at http://127.15.91.3:44775/ using document root <none> and password file <none>
I20260619 18:53:10.810971 18518 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:10.811133 18518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:10.811475 18518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:10.814879 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "abac272f5253476eb22036d1875d5f30"
format_stamp: "Formatted at 2026-06-19 18:53:10 on dist-test-slave-dm3f"
server_key: "426af8c33f93a206dec261340978e167"
server_key_iv: "bd7353d305b53c9f444b88d826af7b3f"
server_key_version: "encryptionkey@0"
I20260619 18:53:10.815825 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "abac272f5253476eb22036d1875d5f30"
format_stamp: "Formatted at 2026-06-19 18:53:10 on dist-test-slave-dm3f"
server_key: "426af8c33f93a206dec261340978e167"
server_key_iv: "bd7353d305b53c9f444b88d826af7b3f"
server_key_version: "encryptionkey@0"
I20260619 18:53:10.821962 18518 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:53:10.826431 18533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:10.828200 18518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:53:10.828435 18518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "abac272f5253476eb22036d1875d5f30"
format_stamp: "Formatted at 2026-06-19 18:53:10 on dist-test-slave-dm3f"
server_key: "426af8c33f93a206dec261340978e167"
server_key_iv: "bd7353d305b53c9f444b88d826af7b3f"
server_key_version: "encryptionkey@0"
I20260619 18:53:10.828689 18518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:10.834930 18378 heartbeater.cc:503] Master 127.15.91.62:42567 was elected leader, sending a full tablet report...
I20260619 18:53:10.861076 18518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:10.866106 18518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:10.866446 18518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:10.867596 18518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:10.869175 18518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:10.869262 18518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:10.869426 18518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:10.869508 18518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:10.906278 18518 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:38151
I20260619 18:53:10.906329 18645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:38151 every 8 connection(s)
I20260619 18:53:10.907775 18518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:53:10.913906 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 18518
I20260619 18:53:10.914124 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:53:10.914587 15724 external_mini_cluster.cc:1428] Setting key 6840d2e915b9882cf4e84b1e2352cb4d
Jun 19 18:53:10 dist-test-slave-dm3f krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895190, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:10.931623 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:10.910641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:59731 (local address 127.15.91.62:42567)
0619 18:53:10.911217 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:10.911221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:10.912383 (+ 1162us) server_negotiation.cc:407] Connection header received
0619 18:53:10.914615 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:10.914621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:10.914765 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:10.914911 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:10.917031 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:10.918268 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:10.920118 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:10.920554 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:10.920674 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:10.924812 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:10.924818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:10.924822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:10.924877 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:10.927866 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:10.928950 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:10.928955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:10.928957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:10.929106 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:10.929631 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:10.929636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:10.929638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:10.929944 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:10.930125 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:10.930956 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 18:53:10.931218 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":323,"threads_started":1}
I20260619 18:53:10.932807 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:10.911336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42567 (local address 127.15.91.3:59731)
0619 18:53:10.912172 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:10.912225 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:10.914249 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:10.915213 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:10.915223 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:10.916028 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:10.916795 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:10.916826 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:10.918445 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:10.918449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:10.919921 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:10.919939 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:10.920862 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:10.920869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:10.921114 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:10.921914 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:10.921950 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:10.924498 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:10.928099 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:10.928109 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:10.928126 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:10.928751 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:10.929242 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:10.929252 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:10.929256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:10.929491 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:10.930087 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:10.930098 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:10.930507 (+ 409us) client_negotiation.cc:769] Sending connection context
0619 18:53:10.930837 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 18:53:10.931471 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":324,"threads_started":1}
I20260619 18:53:10.934706 18646 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42567
I20260619 18:53:10.935161 18646 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:10.936029 18646 heartbeater.cc:511] Master 127.15.91.62:42567 requested a full tablet report, sending...
I20260619 18:53:10.937887 18189 ts_manager.cc:194] Registered new tserver with Master: abac272f5253476eb22036d1875d5f30 (127.15.91.3:38151)
I20260619 18:53:10.938845 18189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:59731
I20260619 18:53:10.943895 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:53:10.977339 18648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:53:10.974771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.15.91.62:42567)
0619 18:53:10.974965 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:10.974970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:10.975459 (+ 489us) server_negotiation.cc:407] Connection header received
0619 18:53:10.975614 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:10.975618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:10.975692 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:10.975802 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:10.977123 (+ 1321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48312: BlockingRecv error: recv got EOF from 127.0.0.1:48312 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:53:10.981030 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 18250
W20260619 18:53:10.984911 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 18250: Not authorized: Client connection negotiation failed: client connection to 127.15.91.1:40567: server requires authentication, but client does not have Kerberos credentials available
I20260619 18:53:10.985306 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 18250
I20260619 18:53:11.003022 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 18384
W20260619 18:53:11.006129 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 18384: Not authorized: Client connection negotiation failed: client connection to 127.15.91.2:37489: server requires authentication, but client does not have Kerberos credentials available
I20260619 18:53:11.006318 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 18384
I20260619 18:53:11.022490 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 18518
W20260619 18:53:11.025928 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 18518: Not authorized: Client connection negotiation failed: client connection to 127.15.91.3:38151: server requires authentication, but client does not have Kerberos credentials available
I20260619 18:53:11.026126 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 18518
I20260619 18:53:11.042639 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 18157
I20260619 18:53:11.044113 18219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:11.151727 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 18157
2026-06-19T18:53:11Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3834 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:53:11 dist-test-slave-dm3f krb5kdc[18664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:53:11 dist-test-slave-dm3f krb5kdc[18664](info): set up 2 sockets
Jun 19 18:53:11 dist-test-slave-dm3f krb5kdc[18664](info): commencing operation
krb5kdc: starting...
W20260619 18:53:12.255354 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:53:12 dist-test-slave-dm3f krb5kdc[18664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:53:12.458586 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38095
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:38095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:12.814427 18677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:12.814812 18677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:12.814913 18677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:12.822625 18677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:12.822793 18677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:12.822854 18677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:12.822914 18677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:12.822962 18677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:12.833395 18677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:38095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:12.834838 18677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:12.836239 18677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:12.849285 18682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:12.849371 18683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:12.849344 18685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:12.850627 18677 server_base.cc:1034] running on GCE node
I20260619 18:53:12.851634 18677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:12.853251 18677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:12.854511 18677 hybrid_clock.cc:648] HybridClock initialized: now 1781895192854444 us; error 46 us; skew 500 ppm
I20260619 18:53:12.855016 18677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:12 dist-test-slave-dm3f krb5kdc[18664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895192, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:12.862512 18677 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:53:12.864748 18677 webserver.cc:460] Webserver started at http://127.15.91.62:33631/ using document root <none> and password file <none>
I20260619 18:53:12.865655 18677 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:12.865835 18677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:12.866191 18677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:12.869725 18677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "0a4af0315f294f08b130c9a85600bb27"
format_stamp: "Formatted at 2026-06-19 18:53:12 on dist-test-slave-dm3f"
server_key: "64263592d968686b713f59fbb39cff60"
server_key_iv: "7921d5373cc36738901468994693305d"
server_key_version: "encryptionkey@0"
I20260619 18:53:12.870777 18677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "0a4af0315f294f08b130c9a85600bb27"
format_stamp: "Formatted at 2026-06-19 18:53:12 on dist-test-slave-dm3f"
server_key: "64263592d968686b713f59fbb39cff60"
server_key_iv: "7921d5373cc36738901468994693305d"
server_key_version: "encryptionkey@0"
I20260619 18:53:12.877097 18677 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:53:12.881742 18692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:12.883639 18677 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:53:12.883919 18677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "0a4af0315f294f08b130c9a85600bb27"
format_stamp: "Formatted at 2026-06-19 18:53:12 on dist-test-slave-dm3f"
server_key: "64263592d968686b713f59fbb39cff60"
server_key_iv: "7921d5373cc36738901468994693305d"
server_key_version: "encryptionkey@0"
I20260619 18:53:12.884179 18677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:12.904482 18677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:12.909448 18677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:12.909808 18677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:12.931329 18677 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:38095
I20260619 18:53:12.931344 18743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:38095 every 8 connection(s)
I20260619 18:53:12.932982 18677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:12.937500 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:12.939100 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 18677
I20260619 18:53:12.939365 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:12.940038 15724 external_mini_cluster.cc:1428] Setting key 4e0c1fb8f34242415b1573d199b6d54a
I20260619 18:53:12.953878 18744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27: Bootstrap starting.
Jun 19 18:53:12 dist-test-slave-dm3f krb5kdc[18664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:12.958495 18744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:12.960187 18744 log.cc:824] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:12.964306 18744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27: No bootstrap required, opened a new log
I20260619 18:53:12.968802 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:12.942177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.15.91.62:38095)
0619 18:53:12.943307 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:12.943322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:12.943370 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:53:12.945302 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:12.945336 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:12.945897 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:12.946302 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:12.948528 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:12.950093 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:12.952263 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:12.953158 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:12.953360 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:12.958769 (+ 5409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:12.958785 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:12.958813 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:12.958878 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:12.963228 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:12.964144 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:12.964155 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:12.964164 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:12.964443 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:12.964909 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:12.964914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:12.964918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:12.965774 (+ 856us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:12.966106 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:12.966597 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 18:53:12.967185 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":593,"threads_started":1}
I20260619 18:53:12.973728 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:53:12.974613 18744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4af0315f294f08b130c9a85600bb27" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38095 } }
I20260619 18:53:12.975214 18744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:12.975312 18744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4af0315f294f08b130c9a85600bb27, State: Initialized, Role: FOLLOWER
I20260619 18:53:12.976083 18744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4af0315f294f08b130c9a85600bb27" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38095 } }
I20260619 18:53:12.976437 18744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:12.976583 18744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:12.976763 18744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:12.980770 18744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4af0315f294f08b130c9a85600bb27" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38095 } }
I20260619 18:53:12.981566 18744 leader_election.cc:302] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a4af0315f294f08b130c9a85600bb27; no voters:
I20260619 18:53:12.982337 18744 leader_election.cc:288] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:12.982587 18750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:12.983808 18750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [term 1 LEADER]: Becoming Leader. State: Replica: 0a4af0315f294f08b130c9a85600bb27, State: Running, Role: LEADER
I20260619 18:53:12.984490 18750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4af0315f294f08b130c9a85600bb27" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38095 } }
I20260619 18:53:12.984952 18744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:12.995051 18751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a4af0315f294f08b130c9a85600bb27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4af0315f294f08b130c9a85600bb27" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38095 } } }
I20260619 18:53:12.995640 18751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:12.996471 18752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a4af0315f294f08b130c9a85600bb27. Latest consensus state: current_term: 1 leader_uuid: "0a4af0315f294f08b130c9a85600bb27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4af0315f294f08b130c9a85600bb27" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38095 } } }
I20260619 18:53:12.996868 18752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:13.003218 18760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:13.009557 18760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:13.023054 18760 catalog_manager.cc:1347] Generated new cluster ID: bad8253e14f1420382e807c2da15d601
I20260619 18:53:13.023191 18760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:13.033350 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:13.022493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.15.91.62:38095)
0619 18:53:13.022755 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:13.022762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:13.023419 (+ 657us) server_negotiation.cc:407] Connection header received
0619 18:53:13.023757 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:13.023764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:13.023855 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:13.023932 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:13.025658 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:13.026837 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:13.028970 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:13.029687 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:13.029845 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:13.031422 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:13.031430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:13.031435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:13.032344 (+ 909us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:13.032451 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:13.032494 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:13.032625 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:13.032847 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 18:53:13.032962 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:53:13.059901 18760 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:13.062036 18760 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:13.073092 18760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a4af0315f294f08b130c9a85600bb27: Generated new TSK 0
I20260619 18:53:13.074106 18760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:53:13.088757 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 18677
I20260619 18:53:13.090128 18739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:13.185006 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 18677
2026-06-19T18:53:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2028 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:53:13 dist-test-slave-dm3f krb5kdc[18779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:53:13 dist-test-slave-dm3f krb5kdc[18779](info): set up 2 sockets
Jun 19 18:53:13 dist-test-slave-dm3f krb5kdc[18779](info): commencing operation
krb5kdc: starting...
W20260619 18:53:14.279313 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:53:14 dist-test-slave-dm3f krb5kdc[18779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:53:14.487210 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:37499
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:37499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:14.841199 18792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:14.841552 18792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:14.841733 18792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:14.849543 18792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:14.849692 18792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:14.849785 18792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:14.849885 18792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:14.849972 18792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:14.860486 18792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:37499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:37499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:14.861999 18792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:14.863490 18792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:14.877004 18797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:14.876971 18800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:14.877055 18798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:14.878000 18792 server_base.cc:1034] running on GCE node
I20260619 18:53:14.878999 18792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:14.880736 18792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:14.882046 18792 hybrid_clock.cc:648] HybridClock initialized: now 1781895194881996 us; error 84 us; skew 500 ppm
I20260619 18:53:14.882507 18792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:14 dist-test-slave-dm3f krb5kdc[18779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895194, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:14.886269 18792 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:53:14.888273 18792 webserver.cc:460] Webserver started at http://127.15.91.62:36977/ using document root <none> and password file <none>
I20260619 18:53:14.889137 18792 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:14.889320 18792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:14.889675 18792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:14.893218 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "a1888308bc5144f0b17e969f6e78b351"
format_stamp: "Formatted at 2026-06-19 18:53:14 on dist-test-slave-dm3f"
server_key: "f645a5ec07b189aa660577f24caa1459"
server_key_iv: "2d839fd48c75189d1ebf3e4d3e333d4e"
server_key_version: "encryptionkey@0"
I20260619 18:53:14.894205 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "a1888308bc5144f0b17e969f6e78b351"
format_stamp: "Formatted at 2026-06-19 18:53:14 on dist-test-slave-dm3f"
server_key: "f645a5ec07b189aa660577f24caa1459"
server_key_iv: "2d839fd48c75189d1ebf3e4d3e333d4e"
server_key_version: "encryptionkey@0"
I20260619 18:53:14.900863 18792 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260619 18:53:14.905364 18807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:14.907963 18792 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:53:14.908344 18792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "a1888308bc5144f0b17e969f6e78b351"
format_stamp: "Formatted at 2026-06-19 18:53:14 on dist-test-slave-dm3f"
server_key: "f645a5ec07b189aa660577f24caa1459"
server_key_iv: "2d839fd48c75189d1ebf3e4d3e333d4e"
server_key_version: "encryptionkey@0"
I20260619 18:53:14.908654 18792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:14.926389 18792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:14.932020 18792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:14.932375 18792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:14.953200 18792 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:37499
I20260619 18:53:14.953219 18858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:37499 every 8 connection(s)
I20260619 18:53:14.954919 18792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:14.958187 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 18792
I20260619 18:53:14.958433 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:14.959178 15724 external_mini_cluster.cc:1428] Setting key dc6f8fc62d9ba3804c2f5dd866803e73
I20260619 18:53:14.959532 18859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:53:14 dist-test-slave-dm3f krb5kdc[18779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:14.976657 18859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351: Bootstrap starting.
I20260619 18:53:14.980446 18859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:14.981729 18859 log.cc:824] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:14.985265 18859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351: No bootstrap required, opened a new log
I20260619 18:53:14.986688 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:14.961625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.15.91.62:37499)
0619 18:53:14.962557 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:14.962570 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:14.962621 (+ 51us) server_negotiation.cc:407] Connection header received
0619 18:53:14.964782 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:14.964813 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:14.965357 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:14.965807 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:14.967856 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:14.969348 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:14.971469 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:14.972412 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:14.972627 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:14.978019 (+ 5392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:14.978030 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:14.978048 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:14.978100 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:14.981722 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:14.982637 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:14.982645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:14.982653 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:14.982939 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:14.983428 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:14.983432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:14.983434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:14.984057 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:14.984422 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:14.984956 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 18:53:14.985528 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":425,"threads_started":1}
I20260619 18:53:14.994024 18859 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1888308bc5144f0b17e969f6e78b351" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37499 } }
I20260619 18:53:14.994498 18859 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:14.994624 18859 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1888308bc5144f0b17e969f6e78b351, State: Initialized, Role: FOLLOWER
I20260619 18:53:14.995447 18859 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1888308bc5144f0b17e969f6e78b351" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37499 } }
I20260619 18:53:14.995738 18859 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:14.995904 18859 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:14.996119 18859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:14.998550 18859 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1888308bc5144f0b17e969f6e78b351" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37499 } }
I20260619 18:53:14.999233 18859 leader_election.cc:302] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1888308bc5144f0b17e969f6e78b351; no voters:
I20260619 18:53:14.999888 18859 leader_election.cc:288] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:15.000057 18864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:15.001207 18864 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [term 1 LEADER]: Becoming Leader. State: Replica: a1888308bc5144f0b17e969f6e78b351, State: Running, Role: LEADER
I20260619 18:53:15.001968 18864 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1888308bc5144f0b17e969f6e78b351" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37499 } }
I20260619 18:53:15.002532 18859 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:15.005323 18866 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1888308bc5144f0b17e969f6e78b351. Latest consensus state: current_term: 1 leader_uuid: "a1888308bc5144f0b17e969f6e78b351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1888308bc5144f0b17e969f6e78b351" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37499 } } }
I20260619 18:53:15.005805 18866 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:15.005342 18865 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1888308bc5144f0b17e969f6e78b351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1888308bc5144f0b17e969f6e78b351" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37499 } } }
I20260619 18:53:15.006597 18865 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:15.008402 18870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:15.014945 18870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:15.025965 18870 catalog_manager.cc:1347] Generated new cluster ID: d394fa80ae024565b621913bb88ec12b
I20260619 18:53:15.026131 18870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:15.034828 18870 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:15.037016 18870 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:15.049800 18870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1888308bc5144f0b17e969f6e78b351: Generated new TSK 0
I20260619 18:53:15.051184 18870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:53:15.111159 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37499
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:15.483992 18885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:15.484369 18885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:15.484546 18885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:15.493080 18885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:15.493265 18885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:15.493436 18885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:53:15.504431 18885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:15.505882 18885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:15.507416 18885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:15.522493 18890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:15.522373 18891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:15.523160 18893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:15.523705 18885 server_base.cc:1034] running on GCE node
I20260619 18:53:15.524396 18885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:15.525489 18885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:15.526799 18885 hybrid_clock.cc:648] HybridClock initialized: now 1781895195526688 us; error 53 us; skew 500 ppm
I20260619 18:53:15.527204 18885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:15 dist-test-slave-dm3f krb5kdc[18779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895195, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:15.531003 18885 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:53:15.532896 18885 webserver.cc:460] Webserver started at http://127.15.91.1:39641/ using document root <none> and password file <none>
I20260619 18:53:15.533680 18885 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:15.533802 18885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:15.534093 18885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:15.537473 18885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "19178826f55d47429bb17e9a8c7a2fc5"
format_stamp: "Formatted at 2026-06-19 18:53:15 on dist-test-slave-dm3f"
server_key: "1b5fcd6193ec6a4dfdd8f6769dde94b0"
server_key_iv: "16611ee84e7107470399fbef78a186c2"
server_key_version: "encryptionkey@0"
I20260619 18:53:15.538374 18885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "19178826f55d47429bb17e9a8c7a2fc5"
format_stamp: "Formatted at 2026-06-19 18:53:15 on dist-test-slave-dm3f"
server_key: "1b5fcd6193ec6a4dfdd8f6769dde94b0"
server_key_iv: "16611ee84e7107470399fbef78a186c2"
server_key_version: "encryptionkey@0"
I20260619 18:53:15.544554 18885 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260619 18:53:15.549335 18900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:15.551517 18885 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:53:15.551857 18885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "19178826f55d47429bb17e9a8c7a2fc5"
format_stamp: "Formatted at 2026-06-19 18:53:15 on dist-test-slave-dm3f"
server_key: "1b5fcd6193ec6a4dfdd8f6769dde94b0"
server_key_iv: "16611ee84e7107470399fbef78a186c2"
server_key_version: "encryptionkey@0"
I20260619 18:53:15.552114 18885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:15.588658 18885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:15.593750 18885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:15.594087 18885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:15.595247 18885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:15.596908 18885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:15.597060 18885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:15.597229 18885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:15.597306 18885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:15.637037 18885 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:32833
I20260619 18:53:15.637104 19012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:32833 every 8 connection(s)
I20260619 18:53:15.638607 18885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:53:15.645102 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 18885
I20260619 18:53:15.645319 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:53:15.645984 15724 external_mini_cluster.cc:1428] Setting key 3175e74bb9c64067d7f2dc5cb7f4be9a
Jun 19 18:53:15 dist-test-slave-dm3f krb5kdc[18779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895195, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:15.664588 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:15.641543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:56035 (local address 127.15.91.62:37499)
0619 18:53:15.642069 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:15.642076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:15.643345 (+ 1269us) server_negotiation.cc:407] Connection header received
0619 18:53:15.645428 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:15.645434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:15.645553 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:15.645679 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:15.647844 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:15.648940 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:15.651240 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:15.652086 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:15.652283 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:15.657808 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:15.657815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:15.657818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:15.657863 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:15.660851 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:15.661885 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:15.661890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:15.661892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:15.662036 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:15.662551 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:15.662556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:15.662558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:15.662932 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:15.663103 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:15.663932 (+ 829us) server_negotiation.cc:299] Negotiation successful
0619 18:53:15.664193 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":247,"threads_started":1}
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:53:15.665711 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:15.642219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:37499 (local address 127.15.91.1:56035)
0619 18:53:15.643132 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:15.643188 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:15.645032 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:15.645986 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:15.645995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:15.646667 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:15.647580 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:15.647609 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:15.649171 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:15.649176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:15.651007 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:15.651030 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:15.652569 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:15.652578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:15.652917 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:15.654010 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:15.654058 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:15.657456 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:15.661079 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:15.661092 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:15.661119 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:15.661704 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:15.662164 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:15.662170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:15.662174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:15.662406 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:15.663078 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:15.663088 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:15.663507 (+ 419us) client_negotiation.cc:769] Sending connection context
0619 18:53:15.663800 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 18:53:15.664425 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":328,"threads_started":1}
I20260619 18:53:15.667779 19013 heartbeater.cc:347] Connected to a master server at 127.15.91.62:37499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:53:15.668167 19013 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:15.669025 19013 heartbeater.cc:511] Master 127.15.91.62:37499 requested a full tablet report, sending...
I20260619 18:53:15.671552 18824 ts_manager.cc:194] Registered new tserver with Master: 19178826f55d47429bb17e9a8c7a2fc5 (127.15.91.1:32833)
I20260619 18:53:15.673924 18824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:56035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:53:15.693861 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37499
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:16.058372 19019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:16.058769 19019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:16.058892 19019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:16.066689 19019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:16.066924 19019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:16.067050 19019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:53:16.077574 19019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:16.079005 19019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:16.080407 19019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:16.095119 19024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:16.095074 19027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:16.095095 19025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:16.096582 19019 server_base.cc:1034] running on GCE node
I20260619 18:53:16.097350 19019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:16.098439 19019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:16.099738 19019 hybrid_clock.cc:648] HybridClock initialized: now 1781895196099643 us; error 91 us; skew 500 ppm
I20260619 18:53:16.100159 19019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:16 dist-test-slave-dm3f krb5kdc[18779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895196, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:16.104068 19019 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:53:16.105935 19019 webserver.cc:460] Webserver started at http://127.15.91.2:42205/ using document root <none> and password file <none>
I20260619 18:53:16.106822 19019 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:16.106997 19019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:16.107342 19019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:16.110868 19019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "152fbeebfa26484ba8909e4fe832b137"
format_stamp: "Formatted at 2026-06-19 18:53:16 on dist-test-slave-dm3f"
server_key: "d2396baef69b61904ddef3d0f15dd07f"
server_key_iv: "46b7791a92d971ecb474e745561d3647"
server_key_version: "encryptionkey@0"
I20260619 18:53:16.111899 19019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "152fbeebfa26484ba8909e4fe832b137"
format_stamp: "Formatted at 2026-06-19 18:53:16 on dist-test-slave-dm3f"
server_key: "d2396baef69b61904ddef3d0f15dd07f"
server_key_iv: "46b7791a92d971ecb474e745561d3647"
server_key_version: "encryptionkey@0"
I20260619 18:53:16.118221 19019 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:53:16.122399 19034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:16.124153 19019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:53:16.124387 19019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "152fbeebfa26484ba8909e4fe832b137"
format_stamp: "Formatted at 2026-06-19 18:53:16 on dist-test-slave-dm3f"
server_key: "d2396baef69b61904ddef3d0f15dd07f"
server_key_iv: "46b7791a92d971ecb474e745561d3647"
server_key_version: "encryptionkey@0"
I20260619 18:53:16.124643 19019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:16.179427 19019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:16.184231 19019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:16.184612 19019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:16.185714 19019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:16.187345 19019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:16.187487 19019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:16.187642 19019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:16.187716 19019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:16.226164 19019 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:35233
I20260619 18:53:16.226230 19146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:35233 every 8 connection(s)
I20260619 18:53:16.227782 19019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:53:16.237414 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 19019
I20260619 18:53:16.237661 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:53:16.238225 15724 external_mini_cluster.cc:1428] Setting key f8134184dcb14bba67f4d9fadb77fa55
Jun 19 18:53:16 dist-test-slave-dm3f krb5kdc[18779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895196, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:16.251801 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:16.231066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:59167 (local address 127.15.91.62:37499)
0619 18:53:16.231536 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:16.231541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:16.232869 (+ 1328us) server_negotiation.cc:407] Connection header received
0619 18:53:16.234867 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:16.234872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:16.234952 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:16.235069 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:16.237015 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.238161 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.240325 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.240765 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.240916 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:16.245184 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:16.245190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:16.245194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:16.245233 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:16.248157 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.249120 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.249125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.249127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.249291 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.249801 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.249806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.249808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.250098 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:16.250263 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:16.251145 (+ 882us) server_negotiation.cc:299] Negotiation successful
0619 18:53:16.251397 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":230,"threads_started":1}
I20260619 18:53:16.252848 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:16.231840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:37499 (local address 127.15.91.2:59167)
0619 18:53:16.232650 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:16.232695 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:16.234467 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:16.235323 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:16.235332 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:16.236007 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:16.236738 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:16.236762 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.238385 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.238395 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:16.240158 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:16.240177 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.241042 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.241051 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:16.241320 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:16.242121 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:16.242147 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:16.244867 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:16.248386 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:16.248397 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:16.248424 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:16.248943 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:16.249429 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:16.249434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:16.249438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:16.249660 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:16.250252 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:16.250262 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:16.250654 (+ 392us) client_negotiation.cc:769] Sending connection context
0619 18:53:16.251018 (+ 364us) client_negotiation.cc:240] Negotiation successful
0619 18:53:16.251636 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":319,"threads_started":1}
I20260619 18:53:16.254815 19147 heartbeater.cc:347] Connected to a master server at 127.15.91.62:37499
I20260619 18:53:16.255183 19147 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:16.256033 19147 heartbeater.cc:511] Master 127.15.91.62:37499 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
I20260619 18:53:16.257989 18824 ts_manager.cc:194] Registered new tserver with Master: 152fbeebfa26484ba8909e4fe832b137 (127.15.91.2:35233)
I20260619 18:53:16.258906 18824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:59167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:53:16.282464 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37499
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:16.647110 19153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:16.647459 19153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:16.647564 19153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:16.655958 19153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:16.656112 19153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:16.656234 19153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:53:16.667070 19153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:16.668517 19153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:16.669898 19153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:53:16.677923 19013 heartbeater.cc:503] Master 127.15.91.62:37499 was elected leader, sending a full tablet report...
W20260619 18:53:16.685218 19158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:16.685261 19159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:16.685170 19161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:16.686811 19153 server_base.cc:1034] running on GCE node
I20260619 18:53:16.687594 19153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:16.688781 19153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:16.690027 19153 hybrid_clock.cc:648] HybridClock initialized: now 1781895196689975 us; error 61 us; skew 500 ppm
I20260619 18:53:16.690366 19153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:16 dist-test-slave-dm3f krb5kdc[18779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895196, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:16.694051 19153 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:53:16.695876 19153 webserver.cc:460] Webserver started at http://127.15.91.3:35893/ using document root <none> and password file <none>
I20260619 18:53:16.696655 19153 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:16.696780 19153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:16.697055 19153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:16.700400 19153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "dbe14ed695bb41cca87903c00984837b"
format_stamp: "Formatted at 2026-06-19 18:53:16 on dist-test-slave-dm3f"
server_key: "6548c804de6d831b1d73a7a0884392eb"
server_key_iv: "a6bf81453247c7d81d86a301d9c5fc64"
server_key_version: "encryptionkey@0"
I20260619 18:53:16.701297 19153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "dbe14ed695bb41cca87903c00984837b"
format_stamp: "Formatted at 2026-06-19 18:53:16 on dist-test-slave-dm3f"
server_key: "6548c804de6d831b1d73a7a0884392eb"
server_key_iv: "a6bf81453247c7d81d86a301d9c5fc64"
server_key_version: "encryptionkey@0"
I20260619 18:53:16.707720 19153 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260619 18:53:16.712231 19168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:16.714309 19153 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260619 18:53:16.714576 19153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "dbe14ed695bb41cca87903c00984837b"
format_stamp: "Formatted at 2026-06-19 18:53:16 on dist-test-slave-dm3f"
server_key: "6548c804de6d831b1d73a7a0884392eb"
server_key_iv: "a6bf81453247c7d81d86a301d9c5fc64"
server_key_version: "encryptionkey@0"
I20260619 18:53:16.714892 19153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:16.738823 19153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:16.743989 19153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:16.744360 19153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:16.745492 19153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:16.747238 19153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:16.747380 19153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:16.747566 19153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:16.747638 19153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:16.785825 19153 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:36729
I20260619 18:53:16.785951 19280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:36729 every 8 connection(s)
I20260619 18:53:16.787377 19153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:53:16.794668 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 19153
I20260619 18:53:16.794935 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:53:16.795411 15724 external_mini_cluster.cc:1428] Setting key 4f62e22ef447a93137598d8aa269b8c1
Jun 19 18:53:16 dist-test-slave-dm3f krb5kdc[18779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895196, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:16.811764 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:16.790442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:43507 (local address 127.15.91.62:37499)
0619 18:53:16.790998 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:16.791003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:16.792175 (+ 1172us) server_negotiation.cc:407] Connection header received
0619 18:53:16.794256 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:16.794263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:16.794366 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:16.794495 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:16.796672 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.797734 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.799608 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.800046 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.800201 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:16.804408 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:16.804414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:16.804418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:16.804458 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:16.807870 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.808850 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.808857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.808861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.809024 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.809569 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.809574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.809577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.809900 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:16.810088 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:16.811065 (+ 977us) server_negotiation.cc:299] Negotiation successful
0619 18:53:16.811362 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":302,"threads_started":1}
I20260619 18:53:16.812819 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:16.791162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:37499 (local address 127.15.91.3:43507)
0619 18:53:16.791982 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:16.792028 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:16.793852 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:16.794852 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:16.794863 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:16.795615 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:16.796465 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:16.796491 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.797928 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.797932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:16.799420 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:16.799439 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.800327 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.800335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:16.800608 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:16.801358 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:16.801405 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:16.804105 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:16.808085 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:16.808097 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:16.808121 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:16.808671 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:16.809158 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:16.809164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:16.809168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:16.809432 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:16.810054 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:16.810063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:16.810560 (+ 497us) client_negotiation.cc:769] Sending connection context
0619 18:53:16.810917 (+ 357us) client_negotiation.cc:240] Negotiation successful
0619 18:53:16.811555 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":312,"threads_started":1}
I20260619 18:53:16.814421 19281 heartbeater.cc:347] Connected to a master server at 127.15.91.62:37499
I20260619 18:53:16.814800 19281 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:16.815585 19281 heartbeater.cc:511] Master 127.15.91.62:37499 requested a full tablet report, sending...
I20260619 18:53:16.817229 18824 ts_manager.cc:194] Registered new tserver with Master: dbe14ed695bb41cca87903c00984837b (127.15.91.3:36729)
I20260619 18:53:16.818063 18824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:43507
I20260619 18:53:16.824630 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:53:16 dist-test-slave-dm3f krb5kdc[18779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895196, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 18:53:16 dist-test-slave-dm3f krb5kdc[18779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895196, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:16.877274 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:16.861266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.15.91.62:37499)
0619 18:53:16.861577 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:16.861586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:16.861989 (+ 403us) server_negotiation.cc:407] Connection header received
0619 18:53:16.862141 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:16.862145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:16.862219 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:16.862314 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:16.863879 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.864635 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.866030 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.866481 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.866602 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:16.871110 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:16.871118 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:16.871121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:16.871167 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:16.874086 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.875082 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.875087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.875089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.875239 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.875702 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.875706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.875708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.876116 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:16.876348 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:16.876709 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 18:53:16.876921 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260619 18:53:16.877975 18824 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34280
I20260619 18:53:16.881160 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 18885
Jun 19 18:53:16 dist-test-slave-dm3f krb5kdc[18779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895196, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:53:16.900027 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:16.882222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.15.91.1:32833)
0619 18:53:16.882915 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:16.882923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:16.882955 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:53:16.883018 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:16.883024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:16.883262 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:16.883414 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:16.884722 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.885798 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.887377 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.888016 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.888144 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:16.893687 (+ 5543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:16.893705 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:16.893722 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:16.893769 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:16.896907 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.897831 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.897840 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.897842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.898005 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.898364 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.898368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.898370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.898831 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:16.899103 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:16.899352 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 18:53:16.899590 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":302,"threads_started":1}
W20260619 18:53:16.901427 19008 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38774
W20260619 18:53:16.902884 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 18885: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:53:16.903144 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 18885
I20260619 18:53:16.920141 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 19019
Jun 19 18:53:16 dist-test-slave-dm3f krb5kdc[18779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895196, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:53:16.939491 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:16.921168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.15.91.2:35233)
0619 18:53:16.921747 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:16.921754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:16.921783 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:53:16.921847 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:16.921853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:16.922086 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:16.922231 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:16.923797 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.924995 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.926526 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.927310 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.927461 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:16.932441 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:16.932458 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:16.932475 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:16.932543 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:16.936017 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.936940 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.936949 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.936952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.937109 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.937553 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.937564 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.937566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.937977 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:16.938340 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:16.938707 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 18:53:16.939080 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":301,"threads_started":1}
W20260619 18:53:16.940999 19142 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39978
W20260619 18:53:16.942343 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 19019: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:53:16.942536 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 19019
I20260619 18:53:16.959452 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 19153
Jun 19 18:53:16 dist-test-slave-dm3f krb5kdc[18779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895196, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:53:16.979696 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:16.960634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.15.91.3:36729)
0619 18:53:16.961465 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:16.961472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:16.961506 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:53:16.961574 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:16.961581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:16.961816 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:16.961972 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:16.963629 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.964808 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.966472 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:16.967492 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:16.967720 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:16.972694 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:16.972715 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:16.972734 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:16.972797 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:16.976305 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.977273 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.977282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.977285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.977485 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:16.977908 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:16.977911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:16.977913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:16.978321 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:16.978587 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:16.979119 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 18:53:16.979376 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":459,"threads_started":1}
W20260619 18:53:16.981184 19276 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45408
W20260619 18:53:16.982591 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 19153: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:53:16.982852 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 19153
I20260619 18:53:16.998497 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 18792
I20260619 18:53:16.999828 18854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:17.112515 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 18792
2026-06-19T18:53:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3933 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:53:17 dist-test-slave-dm3f krb5kdc[19299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:53:17 dist-test-slave-dm3f krb5kdc[19299](info): set up 2 sockets
Jun 19 18:53:17 dist-test-slave-dm3f krb5kdc[19299](info): commencing operation
krb5kdc: starting...
W20260619 18:53:18.219388 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:53:18 dist-test-slave-dm3f krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:53:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:53:18.430261 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38097
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:38097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:18.788055 19312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:18.788491 19312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:18.788667 19312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:18.796555 19312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:18.796754 19312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:18.796869 19312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:18.796977 19312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:18.797060 19312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:18.807744 19312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:38097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:18.809273 19312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:18.810828 19312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:18.824357 19317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:18.824422 19320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:18.824476 19318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:18.825332 19312 server_base.cc:1034] running on GCE node
I20260619 18:53:18.826279 19312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:18.828066 19312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:18.829355 19312 hybrid_clock.cc:648] HybridClock initialized: now 1781895198829307 us; error 62 us; skew 500 ppm
I20260619 18:53:18.829784 19312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:18 dist-test-slave-dm3f krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895198, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:18.836971 19312 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:53:18.838889 19312 webserver.cc:460] Webserver started at http://127.15.91.62:33993/ using document root <none> and password file <none>
I20260619 18:53:18.839707 19312 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:18.839838 19312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:18.840214 19312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:18.843670 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "587eb8c43f5b40cb987d01e98bd53c01"
format_stamp: "Formatted at 2026-06-19 18:53:18 on dist-test-slave-dm3f"
server_key: "954cddac5448f40bc77ed9ad95abe7b0"
server_key_iv: "627c341cca40142e67030d09e5330e49"
server_key_version: "encryptionkey@0"
I20260619 18:53:18.844607 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "587eb8c43f5b40cb987d01e98bd53c01"
format_stamp: "Formatted at 2026-06-19 18:53:18 on dist-test-slave-dm3f"
server_key: "954cddac5448f40bc77ed9ad95abe7b0"
server_key_iv: "627c341cca40142e67030d09e5330e49"
server_key_version: "encryptionkey@0"
I20260619 18:53:18.851037 19312 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260619 18:53:18.855383 19327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:18.857177 19312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:53:18.857466 19312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "587eb8c43f5b40cb987d01e98bd53c01"
format_stamp: "Formatted at 2026-06-19 18:53:18 on dist-test-slave-dm3f"
server_key: "954cddac5448f40bc77ed9ad95abe7b0"
server_key_iv: "627c341cca40142e67030d09e5330e49"
server_key_version: "encryptionkey@0"
I20260619 18:53:18.857693 19312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:18.902407 19312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:18.907339 19312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:18.907693 19312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:18.929157 19312 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:38097
I20260619 18:53:18.929180 19378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:38097 every 8 connection(s)
I20260619 18:53:18.930825 19312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:18.932283 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 19312
I20260619 18:53:18.932557 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:18.933442 15724 external_mini_cluster.cc:1428] Setting key bf66f7867e62de21ed54f387bf81cd9a
I20260619 18:53:18.936538 19379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:53:18 dist-test-slave-dm3f krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:18.951910 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01: Bootstrap starting.
I20260619 18:53:18.955840 19379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:18.957098 19379 log.cc:824] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:18.960397 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:18.935822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.15.91.62:38097)
0619 18:53:18.936829 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:18.936844 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:18.936899 (+ 55us) server_negotiation.cc:407] Connection header received
0619 18:53:18.938783 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:18.938825 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:18.939313 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:18.939754 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:18.941807 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:18.943487 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:18.945453 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:18.946284 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:18.946469 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:18.951661 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:18.951671 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:18.951689 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:18.951741 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:18.955499 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:18.956441 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:18.956450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:18.956457 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:18.956726 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:18.957140 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:18.957145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:18.957149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:18.957829 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:18.958188 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:18.958685 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 18:53:18.959212 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":493,"threads_started":1}
I20260619 18:53:18.960882 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01: No bootstrap required, opened a new log
I20260619 18:53:18.970288 19379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587eb8c43f5b40cb987d01e98bd53c01" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38097 } }
I20260619 18:53:18.970875 19379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:18.971015 19379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587eb8c43f5b40cb987d01e98bd53c01, State: Initialized, Role: FOLLOWER
I20260619 18:53:18.971760 19379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587eb8c43f5b40cb987d01e98bd53c01" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38097 } }
I20260619 18:53:18.972019 19379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:18.972118 19379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:18.972262 19379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:18.974576 19379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587eb8c43f5b40cb987d01e98bd53c01" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38097 } }
I20260619 18:53:18.975333 19379 leader_election.cc:302] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 587eb8c43f5b40cb987d01e98bd53c01; no voters:
I20260619 18:53:18.975998 19379 leader_election.cc:288] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:18.976249 19384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:18.977572 19384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [term 1 LEADER]: Becoming Leader. State: Replica: 587eb8c43f5b40cb987d01e98bd53c01, State: Running, Role: LEADER
I20260619 18:53:18.978215 19384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587eb8c43f5b40cb987d01e98bd53c01" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38097 } }
I20260619 18:53:18.978614 19379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:18.981339 19386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 587eb8c43f5b40cb987d01e98bd53c01. Latest consensus state: current_term: 1 leader_uuid: "587eb8c43f5b40cb987d01e98bd53c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587eb8c43f5b40cb987d01e98bd53c01" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38097 } } }
I20260619 18:53:18.981767 19386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:18.982362 19385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "587eb8c43f5b40cb987d01e98bd53c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587eb8c43f5b40cb987d01e98bd53c01" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38097 } } }
I20260619 18:53:18.982805 19385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:18.985133 19392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:18.990976 19392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:19.002684 19392 catalog_manager.cc:1347] Generated new cluster ID: cac14e9b33ce4e2591bbf6c307f6e6a7
I20260619 18:53:19.002866 19392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:19.024438 19392 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:19.026117 19392 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:19.042496 19392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 587eb8c43f5b40cb987d01e98bd53c01: Generated new TSK 0
I20260619 18:53:19.043555 19392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:53:19.103427 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38097
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:19.469692 19405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:19.470119 19405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:19.470297 19405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:19.478273 19405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:19.478487 19405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:19.478664 19405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:53:19.490219 19405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:19.492213 19405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:19.493974 19405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:19.509491 19410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:19.509544 19413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:19.509516 19411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:19.510689 19405 server_base.cc:1034] running on GCE node
I20260619 18:53:19.511533 19405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:19.512715 19405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:19.513988 19405 hybrid_clock.cc:648] HybridClock initialized: now 1781895199513922 us; error 47 us; skew 500 ppm
I20260619 18:53:19.514478 19405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:19 dist-test-slave-dm3f krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895199, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:19.518532 19405 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:53:19.520442 19405 webserver.cc:460] Webserver started at http://127.15.91.1:35095/ using document root <none> and password file <none>
I20260619 18:53:19.521306 19405 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:19.521499 19405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:19.521863 19405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:19.525338 19405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "1170134b68594332a2afb68bb5fc4b5f"
format_stamp: "Formatted at 2026-06-19 18:53:19 on dist-test-slave-dm3f"
server_key: "928ba33cb1d5a18171653b240e2ce410"
server_key_iv: "b2a6051ba631d4b1e839eee3d01ce825"
server_key_version: "encryptionkey@0"
I20260619 18:53:19.526360 19405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "1170134b68594332a2afb68bb5fc4b5f"
format_stamp: "Formatted at 2026-06-19 18:53:19 on dist-test-slave-dm3f"
server_key: "928ba33cb1d5a18171653b240e2ce410"
server_key_iv: "b2a6051ba631d4b1e839eee3d01ce825"
server_key_version: "encryptionkey@0"
I20260619 18:53:19.532724 19405 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 18:53:19.537293 19420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:19.539182 19405 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:53:19.539438 19405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "1170134b68594332a2afb68bb5fc4b5f"
format_stamp: "Formatted at 2026-06-19 18:53:19 on dist-test-slave-dm3f"
server_key: "928ba33cb1d5a18171653b240e2ce410"
server_key_iv: "b2a6051ba631d4b1e839eee3d01ce825"
server_key_version: "encryptionkey@0"
I20260619 18:53:19.539710 19405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:19.563778 19405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:19.569116 19405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:19.569502 19405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:19.570655 19405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:19.572355 19405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:19.572504 19405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:19.572666 19405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:19.572748 19405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:19.610893 19405 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:43331
I20260619 18:53:19.610994 19532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:43331 every 8 connection(s)
I20260619 18:53:19.612365 19405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:53:19.617478 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 19405
I20260619 18:53:19.617719 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:53:19.618577 15724 external_mini_cluster.cc:1428] Setting key b8a189169bff8bab5b4f110e2406ce3a
Jun 19 18:53:19 dist-test-slave-dm3f krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895199, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:19.638533 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:19.615365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:42763 (local address 127.15.91.62:38097)
0619 18:53:19.615862 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:19.615867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:19.616905 (+ 1038us) server_negotiation.cc:407] Connection header received
0619 18:53:19.619143 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:19.619171 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:19.619306 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:19.619450 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:19.621513 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:19.622244 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:19.624144 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:19.624579 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:19.624755 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:19.629917 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:19.629929 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:19.629938 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:19.629998 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:19.634159 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:19.635438 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:19.635449 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:19.635456 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:19.635665 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:19.636271 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:19.636285 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:19.636291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:19.636764 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:19.636975 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:19.637795 (+ 820us) server_negotiation.cc:299] Negotiation successful
0619 18:53:19.638104 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":280,"threads_started":1}
I20260619 18:53:19.639443 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:19.615859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38097 (local address 127.15.91.1:42763)
0619 18:53:19.616712 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:19.616760 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:19.618783 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:19.619826 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:19.619843 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:19.620600 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:19.621282 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:19.621303 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:19.622460 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:19.622464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:19.623922 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:19.623941 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:19.624869 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:19.624878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:19.625273 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:19.626254 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:19.626289 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:19.629533 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:19.634430 (+ 4897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:19.634440 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:19.634456 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:19.635224 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:19.635839 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:19.635843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:19.635846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:19.636079 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:19.636918 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:19.636925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:19.637311 (+ 386us) client_negotiation.cc:769] Sending connection context
0619 18:53:19.637576 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 18:53:19.638041 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":345,"threads_started":1}
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:53:19.641834 19533 heartbeater.cc:347] Connected to a master server at 127.15.91.62:38097
I20260619 18:53:19.642241 19533 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:19.643167 19533 heartbeater.cc:511] Master 127.15.91.62:38097 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:53:19.646195 19344 ts_manager.cc:194] Registered new tserver with Master: 1170134b68594332a2afb68bb5fc4b5f (127.15.91.1:43331)
I20260619 18:53:19.647892 19344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:42763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:53:19.670972 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38097
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:20.054550 19539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:20.055034 19539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:20.055222 19539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:20.063526 19539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:20.063706 19539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:20.063901 19539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:53:20.074404 19539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:20.075922 19539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:20.077414 19539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:20.092492 19544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:20.092599 19547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:20.092561 19545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:20.093778 19539 server_base.cc:1034] running on GCE node
I20260619 18:53:20.094628 19539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:20.095778 19539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:20.097059 19539 hybrid_clock.cc:648] HybridClock initialized: now 1781895200097003 us; error 53 us; skew 500 ppm
I20260619 18:53:20.097445 19539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:20 dist-test-slave-dm3f krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895200, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:20.101100 19539 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:53:20.102943 19539 webserver.cc:460] Webserver started at http://127.15.91.2:33043/ using document root <none> and password file <none>
I20260619 18:53:20.103838 19539 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:20.104008 19539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:20.104306 19539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:20.107594 19539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "5d23fedc40f340f7a2cb5bfa1e218a92"
format_stamp: "Formatted at 2026-06-19 18:53:20 on dist-test-slave-dm3f"
server_key: "fcc2dca9c4eaa568968990a2719d4308"
server_key_iv: "e661a055882b7e00f92bcb2a464e0d73"
server_key_version: "encryptionkey@0"
I20260619 18:53:20.108484 19539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "5d23fedc40f340f7a2cb5bfa1e218a92"
format_stamp: "Formatted at 2026-06-19 18:53:20 on dist-test-slave-dm3f"
server_key: "fcc2dca9c4eaa568968990a2719d4308"
server_key_iv: "e661a055882b7e00f92bcb2a464e0d73"
server_key_version: "encryptionkey@0"
I20260619 18:53:20.114611 19539 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 18:53:20.118860 19554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:20.120610 19539 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260619 18:53:20.120810 19539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "5d23fedc40f340f7a2cb5bfa1e218a92"
format_stamp: "Formatted at 2026-06-19 18:53:20 on dist-test-slave-dm3f"
server_key: "fcc2dca9c4eaa568968990a2719d4308"
server_key_iv: "e661a055882b7e00f92bcb2a464e0d73"
server_key_version: "encryptionkey@0"
I20260619 18:53:20.121019 19539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:20.139705 19539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:20.145102 19539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:20.145383 19539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:20.146391 19539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:20.148065 19539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:20.148157 19539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:20.148259 19539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:20.148304 19539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:20.186275 19539 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:42461
I20260619 18:53:20.186338 19666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:42461 every 8 connection(s)
I20260619 18:53:20.187726 19539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:53:20.193339 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 19539
I20260619 18:53:20.193557 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:53:20.194000 15724 external_mini_cluster.cc:1428] Setting key d6e8f683eec08f42bca3ba885bb76922
Jun 19 18:53:20 dist-test-slave-dm3f krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895200, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:20.213723 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:20.190658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:45751 (local address 127.15.91.62:38097)
0619 18:53:20.191170 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:20.191175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:20.192402 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 18:53:20.194856 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:20.194864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:20.194965 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:20.195119 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:20.197461 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.198830 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.200704 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.201556 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.201749 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:20.206807 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:20.206813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:20.206817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:20.206860 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:20.209750 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:20.210933 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:20.210945 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:20.210949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:20.211122 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:20.211697 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:20.211702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:20.211704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:20.212001 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:20.212181 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:20.213024 (+ 843us) server_negotiation.cc:299] Negotiation successful
0619 18:53:20.213311 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":258,"threads_started":1}
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
I20260619 18:53:20.214915 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:20.191342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38097 (local address 127.15.91.2:45751)
0619 18:53:20.192195 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:20.192243 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:20.194347 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:20.195502 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:20.195515 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:20.196471 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:20.197200 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:20.197223 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.199055 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.199060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:20.200462 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:20.200482 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.201851 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.201857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:20.202263 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:20.203230 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:20.203270 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:20.206398 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:20.209991 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:20.210003 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:20.210059 (+ 56us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:20.210685 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:20.211283 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:20.211288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:20.211292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:20.211538 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:20.212175 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:20.212185 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:20.212614 (+ 429us) client_negotiation.cc:769] Sending connection context
0619 18:53:20.212920 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 18:53:20.213531 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":344,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
I20260619 18:53:20.216887 19667 heartbeater.cc:347] Connected to a master server at 127.15.91.62:38097
I20260619 18:53:20.217316 19667 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:20.218446 19667 heartbeater.cc:511] Master 127.15.91.62:38097 requested a full tablet report, sending...
I20260619 18:53:20.220711 19344 ts_manager.cc:194] Registered new tserver with Master: 5d23fedc40f340f7a2cb5bfa1e218a92 (127.15.91.2:42461)
I20260619 18:53:20.222155 19344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:45751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:53:20.244356 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38097
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:20.603137 19673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:20.603456 19673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:20.603659 19673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:20.611570 19673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:20.611747 19673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:20.611944 19673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:53:20.622782 19673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:20.624245 19673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:20.625816 19673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:20.642359 19679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:20.642481 19678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:20.642458 19681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:20.643911 19673 server_base.cc:1034] running on GCE node
I20260619 18:53:20.644758 19673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:20.646924 19673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:20.648200 19673 hybrid_clock.cc:648] HybridClock initialized: now 1781895200648149 us; error 62 us; skew 500 ppm
I20260619 18:53:20.648620 19673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:53:20.651710 19533 heartbeater.cc:503] Master 127.15.91.62:38097 was elected leader, sending a full tablet report...
Jun 19 18:53:20 dist-test-slave-dm3f krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895200, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:20.652773 19673 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:53:20.654908 19673 webserver.cc:460] Webserver started at http://127.15.91.3:45205/ using document root <none> and password file <none>
I20260619 18:53:20.655751 19673 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:20.655895 19673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:20.656252 19673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:20.659840 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "8508092921b04b9a9caeec3a32ed46e8"
format_stamp: "Formatted at 2026-06-19 18:53:20 on dist-test-slave-dm3f"
server_key: "fdbfac8b9253aeb46ecb5a390f6e72f9"
server_key_iv: "b2461b7bebd43dcfc0f69aefb1217069"
server_key_version: "encryptionkey@0"
I20260619 18:53:20.660866 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "8508092921b04b9a9caeec3a32ed46e8"
format_stamp: "Formatted at 2026-06-19 18:53:20 on dist-test-slave-dm3f"
server_key: "fdbfac8b9253aeb46ecb5a390f6e72f9"
server_key_iv: "b2461b7bebd43dcfc0f69aefb1217069"
server_key_version: "encryptionkey@0"
I20260619 18:53:20.668183 19673 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260619 18:53:20.672595 19688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:20.674578 19673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:53:20.674906 19673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "8508092921b04b9a9caeec3a32ed46e8"
format_stamp: "Formatted at 2026-06-19 18:53:20 on dist-test-slave-dm3f"
server_key: "fdbfac8b9253aeb46ecb5a390f6e72f9"
server_key_iv: "b2461b7bebd43dcfc0f69aefb1217069"
server_key_version: "encryptionkey@0"
I20260619 18:53:20.675190 19673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:20.695801 19673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:20.701092 19673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:20.701449 19673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:20.702577 19673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:20.704351 19673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:20.704499 19673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:20.704675 19673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:20.704756 19673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:20.747341 19673 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:46637
I20260619 18:53:20.747382 19800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:46637 every 8 connection(s)
I20260619 18:53:20.749502 19673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:53:20.756979 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 19673
I20260619 18:53:20.757236 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:53:20.757747 15724 external_mini_cluster.cc:1428] Setting key d79586a1b879849e44e17013254458d3
Jun 19 18:53:20 dist-test-slave-dm3f krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895200, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:20.774618 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:20.752749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:50327 (local address 127.15.91.62:38097)
0619 18:53:20.753305 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:20.753310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:20.754470 (+ 1160us) server_negotiation.cc:407] Connection header received
0619 18:53:20.756781 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:20.756790 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:20.756898 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:20.757049 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:20.759412 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.760251 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.762041 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.762525 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.762680 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:20.766886 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:20.766892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:20.766895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:20.766936 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:20.770414 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:20.771640 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:20.771646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:20.771649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:20.771809 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:20.772392 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:20.772397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:20.772399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:20.772767 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:20.772960 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:20.773855 (+ 895us) server_negotiation.cc:299] Negotiation successful
0619 18:53:20.774146 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":282,"threads_started":1}
I20260619 18:53:20.775854 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:20.753416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38097 (local address 127.15.91.3:50327)
0619 18:53:20.754256 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:20.754309 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:20.756368 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:20.757442 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:20.757452 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:20.758170 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:20.759181 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:20.759221 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.760449 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.760456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:20.761839 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:20.761855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.762834 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.762842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:20.763271 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:20.764056 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:20.764090 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:20.766531 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:20.770693 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:20.770704 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:20.770796 (+ 92us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:20.771428 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:20.771956 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:20.771962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:20.771967 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:20.772224 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:20.772931 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:20.772942 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:20.773385 (+ 443us) client_negotiation.cc:769] Sending connection context
0619 18:53:20.773712 (+ 327us) client_negotiation.cc:240] Negotiation successful
0619 18:53:20.774383 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":322,"threads_started":1}
I20260619 18:53:20.778167 19801 heartbeater.cc:347] Connected to a master server at 127.15.91.62:38097
I20260619 18:53:20.778566 19801 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:20.779661 19801 heartbeater.cc:511] Master 127.15.91.62:38097 requested a full tablet report, sending...
I20260619 18:53:20.781504 19344 ts_manager.cc:194] Registered new tserver with Master: 8508092921b04b9a9caeec3a32ed46e8 (127.15.91.3:46637)
I20260619 18:53:20.782397 19344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:50327
I20260619 18:53:20.787361 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:53:20 dist-test-slave-dm3f krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:53:20 dist-test-slave-dm3f krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:53:20.844800 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:20.826646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.15.91.1:43331)
0619 18:53:20.827860 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:20.827869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:20.827898 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:53:20.827965 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:20.827971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:20.828211 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:20.828378 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:20.830029 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.831238 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.832888 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.833614 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.833754 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:20.838349 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:20.838385 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:20.838402 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:20.838449 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:20.841532 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:20.842390 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:20.842399 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:20.842401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:20.842549 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:20.843004 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:20.843008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:20.843010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:20.843401 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:20.843659 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:20.844094 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 18:53:20.844408 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":890,"threads_started":1}
Jun 19 18:53:20 dist-test-slave-dm3f krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:20.866053 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:20.850317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.15.91.62:38097)
0619 18:53:20.850598 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:20.850604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:20.850829 (+ 225us) server_negotiation.cc:407] Connection header received
0619 18:53:20.851068 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:20.851074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:20.851201 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:20.851323 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:20.852704 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.853465 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.854916 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.855362 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.855495 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:20.860110 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:20.860115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:20.860119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:20.860164 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:20.863129 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:20.864059 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:20.864063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:20.864066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:20.864220 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:20.864613 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:20.864617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:20.864619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:20.864967 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:20.865173 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:20.865601 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 18:53:20.865807 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260619 18:53:20.866678 19344 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41106
I20260619 18:53:20.868012 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 19405
I20260619 18:53:20.882947 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:20.869106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.15.91.1:43331)
0619 18:53:20.869354 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:20.869360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:20.869448 (+ 88us) server_negotiation.cc:407] Connection header received
0619 18:53:20.869597 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:20.869602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:20.869708 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:20.869805 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:20.871272 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.872003 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.873428 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.873907 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.874021 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:20.876896 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:20.876902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:20.876905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:20.876950 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:20.880107 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:20.880986 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:20.880991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:20.880993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:20.881138 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:20.881581 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:20.881585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:20.881587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:20.881859 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:20.882052 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:20.882386 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 18:53:20.882578 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 18:53:20.883760 19528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:20.968377 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 19405
I20260619 18:53:20.985913 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 19539
Jun 19 18:53:20 dist-test-slave-dm3f krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:53:21.005088 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:20.987058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.15.91.2:42461)
0619 18:53:20.987704 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:20.987712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:20.987740 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:53:20.987805 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:20.987812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:20.988043 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:20.988193 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:20.989499 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.990575 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.992089 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:20.992729 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:20.992867 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:20.997662 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:20.997680 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:20.997697 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:20.997745 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:21.001031 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:21.002006 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:21.002015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:21.002017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:21.002176 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:21.002866 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:21.002873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:21.002876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:21.003516 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:21.003912 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:21.004452 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 18:53:21.004743 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":374,"threads_started":1}
I20260619 18:53:21.006821 19662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:21.094305 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 19539
I20260619 18:53:21.110888 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 19673
Jun 19 18:53:21 dist-test-slave-dm3f krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:53:21.130090 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:21.112154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.15.91.3:46637)
0619 18:53:21.112856 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:21.112864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:21.112897 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:53:21.112969 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:21.112976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:21.113212 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:21.113377 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:21.114832 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:21.115953 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:21.117472 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:21.118122 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:21.118257 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:21.123413 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:21.123432 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:21.123450 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:21.123497 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:21.126863 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:21.127787 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:21.127796 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:21.127798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:21.127970 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:21.128363 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:21.128367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:21.128369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:21.128765 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:21.129055 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:21.129377 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 18:53:21.129624 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":392,"threads_started":1}
I20260619 18:53:21.131705 19796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:21.217784 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 19673
I20260619 18:53:21.234515 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 19312
I20260619 18:53:21.236117 19374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:21.345266 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 19312
2026-06-19T18:53:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4235 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:53:21 dist-test-slave-dm3f krb5kdc[19821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:53:21 dist-test-slave-dm3f krb5kdc[19821](info): set up 2 sockets
Jun 19 18:53:21 dist-test-slave-dm3f krb5kdc[19821](info): commencing operation
krb5kdc: starting...
W20260619 18:53:22.471724 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:53:22 dist-test-slave-dm3f krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:53:22.687263 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:36317
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:36317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:23.039247 19834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:23.039597 19834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:23.039772 19834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:23.047740 19834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:23.047881 19834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:23.047974 19834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:23.048064 19834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:23.048177 19834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:23.058970 19834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:36317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:23.060464 19834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:23.062003 19834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:23.075780 19842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:23.075750 19839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:23.075747 19840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:23.076824 19834 server_base.cc:1034] running on GCE node
I20260619 18:53:23.077725 19834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:23.079530 19834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:23.080853 19834 hybrid_clock.cc:648] HybridClock initialized: now 1781895203080796 us; error 74 us; skew 500 ppm
I20260619 18:53:23.081324 19834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:23 dist-test-slave-dm3f krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895203, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:23.085045 19834 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:53:23.086504 19834 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:23.086694 19834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:23.087136 19834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:23.090577 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "81e93974ad224197966ff741a3bd86c1"
format_stamp: "Formatted at 2026-06-19 18:53:23 on dist-test-slave-dm3f"
server_key: "539f6b566a3c8fa2e6f6f1e7d96f365c"
server_key_iv: "6af0a519939e325fee677358f6315f55"
server_key_version: "encryptionkey@0"
I20260619 18:53:23.091683 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "81e93974ad224197966ff741a3bd86c1"
format_stamp: "Formatted at 2026-06-19 18:53:23 on dist-test-slave-dm3f"
server_key: "539f6b566a3c8fa2e6f6f1e7d96f365c"
server_key_iv: "6af0a519939e325fee677358f6315f55"
server_key_version: "encryptionkey@0"
I20260619 18:53:23.097839 19834 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:53:23.102072 19848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:23.103767 19834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:53:23.104058 19834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "81e93974ad224197966ff741a3bd86c1"
format_stamp: "Formatted at 2026-06-19 18:53:23 on dist-test-slave-dm3f"
server_key: "539f6b566a3c8fa2e6f6f1e7d96f365c"
server_key_iv: "6af0a519939e325fee677358f6315f55"
server_key_version: "encryptionkey@0"
I20260619 18:53:23.104342 19834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:23.131835 19834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:23.136668 19834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:23.136986 19834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:23.158022 19834 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:36317
I20260619 18:53:23.158041 19899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:36317 every 8 connection(s)
I20260619 18:53:23.159200 19834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:23.163373 19900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:23.169711 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 19834
I20260619 18:53:23.169962 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:23.170807 15724 external_mini_cluster.cc:1428] Setting key 79b5417c4016a588ccdcdbcdf3451c76
I20260619 18:53:23.176160 19900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1: Bootstrap starting.
I20260619 18:53:23.179854 19900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:23.181146 19900 log.cc:824] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:23.184679 19900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1: No bootstrap required, opened a new log
Jun 19 18:53:23 dist-test-slave-dm3f krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:23.193815 19900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e93974ad224197966ff741a3bd86c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36317 } }
I20260619 18:53:23.194293 19900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:23.194417 19900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e93974ad224197966ff741a3bd86c1, State: Initialized, Role: FOLLOWER
I20260619 18:53:23.195308 19900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e93974ad224197966ff741a3bd86c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36317 } }
I20260619 18:53:23.195613 19900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:23.195767 19900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:23.195989 19900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:23.196905 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:23.173017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.15.91.62:36317)
0619 18:53:23.173916 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:23.173930 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:23.173987 (+ 57us) server_negotiation.cc:407] Connection header received
0619 18:53:23.175962 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:23.176004 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:23.176612 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:23.177093 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:23.178802 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:23.180337 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:23.181854 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:23.182699 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:23.183007 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:23.187696 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:23.187708 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:23.187727 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:23.187777 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:23.191495 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:23.192406 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:23.192414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:23.192423 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:23.192655 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:23.193114 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:23.193122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:23.193129 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:23.193821 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:23.194198 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:23.194742 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 18:53:23.195249 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":368,"threads_started":1}
I20260619 18:53:23.198978 19900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e93974ad224197966ff741a3bd86c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36317 } }
I20260619 18:53:23.199586 19900 leader_election.cc:302] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e93974ad224197966ff741a3bd86c1; no voters:
I20260619 18:53:23.200304 19900 leader_election.cc:288] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:23.200433 19905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:23.201164 19905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [term 1 LEADER]: Becoming Leader. State: Replica: 81e93974ad224197966ff741a3bd86c1, State: Running, Role: LEADER
I20260619 18:53:23.201889 19905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e93974ad224197966ff741a3bd86c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36317 } }
I20260619 18:53:23.203039 19900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:23.205211 19906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e93974ad224197966ff741a3bd86c1. Latest consensus state: current_term: 1 leader_uuid: "81e93974ad224197966ff741a3bd86c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e93974ad224197966ff741a3bd86c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36317 } } }
I20260619 18:53:23.205654 19906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:23.205364 19907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81e93974ad224197966ff741a3bd86c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e93974ad224197966ff741a3bd86c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36317 } } }
I20260619 18:53:23.205860 19907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:23.211998 19912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:23.218513 19912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:23.228791 19912 catalog_manager.cc:1347] Generated new cluster ID: f9aa9aefe0a24880bea9a6bcc36f7ece
I20260619 18:53:23.228951 19912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:23.257367 19912 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:23.259107 19912 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:23.268061 19912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1: Generated new TSK 0
I20260619 18:53:23.269150 19912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:53:23.330688 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36317
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:23.735255 19926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:23.735620 19926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:23.735742 19926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:23.743854 19926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:23.744009 19926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:23.744134 19926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:53:23.754817 19926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:23.756227 19926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:23.757753 19926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:23.772917 19931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:23.772989 19934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:23.772985 19932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:23.774803 19926 server_base.cc:1034] running on GCE node
I20260619 18:53:23.775588 19926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:23.776783 19926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:23.778067 19926 hybrid_clock.cc:648] HybridClock initialized: now 1781895203777992 us; error 79 us; skew 500 ppm
I20260619 18:53:23.778472 19926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:23 dist-test-slave-dm3f krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895203, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:23.782213 19926 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:53:23.783648 19926 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:23.783766 19926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:23.784068 19926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:23.787556 19926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "01e8289b6b0e4c2f8db033600b080198"
format_stamp: "Formatted at 2026-06-19 18:53:23 on dist-test-slave-dm3f"
server_key: "cc6a5120eff564831808b8a1b782c1a7"
server_key_iv: "0817529139ff6071c9569eb8e1ee44bb"
server_key_version: "encryptionkey@0"
I20260619 18:53:23.788506 19926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "01e8289b6b0e4c2f8db033600b080198"
format_stamp: "Formatted at 2026-06-19 18:53:23 on dist-test-slave-dm3f"
server_key: "cc6a5120eff564831808b8a1b782c1a7"
server_key_iv: "0817529139ff6071c9569eb8e1ee44bb"
server_key_version: "encryptionkey@0"
I20260619 18:53:23.794569 19926 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 18:53:23.799144 19940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:23.800860 19926 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260619 18:53:23.801105 19926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "01e8289b6b0e4c2f8db033600b080198"
format_stamp: "Formatted at 2026-06-19 18:53:23 on dist-test-slave-dm3f"
server_key: "cc6a5120eff564831808b8a1b782c1a7"
server_key_iv: "0817529139ff6071c9569eb8e1ee44bb"
server_key_version: "encryptionkey@0"
I20260619 18:53:23.801367 19926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:23.842564 19926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:23.847812 19926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:23.848145 19926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:23.849293 19926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:23.851105 19926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:23.851223 19926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:23.851375 19926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:23.851475 19926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:23.890841 19926 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:42385
I20260619 18:53:23.890952 20052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:42385 every 8 connection(s)
I20260619 18:53:23.891870 19926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:53:23.897979 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 19926
I20260619 18:53:23.898218 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:53:23.899080 15724 external_mini_cluster.cc:1428] Setting key e6407b0ac5df4ea93222928b9da8eb8d
Jun 19 18:53:23 dist-test-slave-dm3f krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895203, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:23.920486 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:23.894924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:41783 (local address 127.15.91.62:36317)
0619 18:53:23.895433 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:23.895437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:23.896509 (+ 1072us) server_negotiation.cc:407] Connection header received
0619 18:53:23.899340 (+ 2831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:23.899346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:23.899432 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:23.899546 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:23.901907 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:23.902664 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:23.905799 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:23.906387 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:23.906530 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:23.912049 (+ 5519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:23.912059 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:23.912067 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:23.912127 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:23.916162 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:23.917268 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:23.917278 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:23.917284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:23.917478 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:23.918103 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:23.918110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:23.918116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:23.918574 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:23.918826 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:23.919703 (+ 877us) server_negotiation.cc:299] Negotiation successful
0619 18:53:23.920030 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":282,"threads_started":1}
I20260619 18:53:23.921365 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:23.895440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:36317 (local address 127.15.91.1:41783)
0619 18:53:23.896262 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:23.896333 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:23.898890 (+ 2557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:23.899905 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:23.899916 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:23.900697 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:23.901657 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:23.901694 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:23.902938 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:23.902948 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:23.905470 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:23.905501 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:23.906698 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:23.906709 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:23.907170 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:23.908283 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:23.908325 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:23.911686 (+ 3361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:23.916461 (+ 4775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:23.916472 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:23.916490 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:23.917082 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:23.917671 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:23.917678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:23.917682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:23.917949 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:23.918811 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:23.918819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:23.919226 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 18:53:23.919529 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 18:53:23.920008 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":337,"threads_started":1}
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:53:23.923564 20053 heartbeater.cc:347] Connected to a master server at 127.15.91.62:36317
I20260619 18:53:23.923930 20053 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:53:23.924791 20053 heartbeater.cc:511] Master 127.15.91.62:36317 requested a full tablet report, sending...
I20260619 18:53:23.927556 19865 ts_manager.cc:194] Registered new tserver with Master: 01e8289b6b0e4c2f8db033600b080198 (127.15.91.1:42385)
I20260619 18:53:23.929193 19865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:41783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:53:23.950582 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36317
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:24.310698 20059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:24.311087 20059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:24.311198 20059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:24.318948 20059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:24.319092 20059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:24.319212 20059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:53:24.329478 20059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:24.330981 20059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:24.332348 20059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:24.351405 20067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:24.351485 20065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:24.351440 20064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:24.352372 20059 server_base.cc:1034] running on GCE node
I20260619 18:53:24.353118 20059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:24.354219 20059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:24.355484 20059 hybrid_clock.cc:648] HybridClock initialized: now 1781895204355435 us; error 63 us; skew 500 ppm
I20260619 18:53:24.355870 20059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:24 dist-test-slave-dm3f krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895204, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:24.359563 20059 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:53:24.361048 20059 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:24.361222 20059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:24.361598 20059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:24.365018 20059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927"
format_stamp: "Formatted at 2026-06-19 18:53:24 on dist-test-slave-dm3f"
server_key: "b74e23332d8524bead1be5f28bda6d83"
server_key_iv: "a76226c2a9e4ffed6569eb9ac6d6cb7c"
server_key_version: "encryptionkey@0"
I20260619 18:53:24.366042 20059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927"
format_stamp: "Formatted at 2026-06-19 18:53:24 on dist-test-slave-dm3f"
server_key: "b74e23332d8524bead1be5f28bda6d83"
server_key_iv: "a76226c2a9e4ffed6569eb9ac6d6cb7c"
server_key_version: "encryptionkey@0"
I20260619 18:53:24.372155 20059 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260619 18:53:24.376482 20073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:24.378136 20059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 18:53:24.378315 20059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927"
format_stamp: "Formatted at 2026-06-19 18:53:24 on dist-test-slave-dm3f"
server_key: "b74e23332d8524bead1be5f28bda6d83"
server_key_iv: "a76226c2a9e4ffed6569eb9ac6d6cb7c"
server_key_version: "encryptionkey@0"
I20260619 18:53:24.378531 20059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:24.400482 20059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:24.405848 20059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:24.406123 20059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:24.407284 20059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:24.408999 20059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:24.409118 20059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:24.409205 20059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:24.409260 20059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:24.447746 20059 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:41469
I20260619 18:53:24.447798 20185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:41469 every 8 connection(s)
I20260619 18:53:24.448788 20059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:53:24.453209 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 20059
I20260619 18:53:24.453449 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:53:24.454017 15724 external_mini_cluster.cc:1428] Setting key 9d64091907af0e948731cfd8a1f047a9
Jun 19 18:53:24 dist-test-slave-dm3f krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895204, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
I20260619 18:53:24.475293 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:24.451907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:40763 (local address 127.15.91.62:36317)
0619 18:53:24.452401 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:24.452407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:24.453742 (+ 1335us) server_negotiation.cc:407] Connection header received
0619 18:53:24.456337 (+ 2595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:24.456342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:24.456467 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:24.456610 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:24.458480 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:24.459729 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:24.461703 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:24.462267 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:24.462396 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:24.467853 (+ 5457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:24.467859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:24.467863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:24.467907 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:24.471068 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:24.472048 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:24.472056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:24.472065 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:24.472218 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:24.472849 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:24.472856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:24.472859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:24.473386 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:24.473586 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:24.474486 (+ 900us) server_negotiation.cc:299] Negotiation successful
0619 18:53:24.474837 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":237,"threads_started":1}
I20260619 18:53:24.476189 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:24.452544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:36317 (local address 127.15.91.2:40763)
0619 18:53:24.453523 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:24.453578 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:24.455926 (+ 2348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:24.456871 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:24.456881 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:24.457517 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:24.458200 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:24.458222 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:24.459984 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:24.459989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:24.461430 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:24.461460 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:24.462645 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:24.462656 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:24.463117 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:24.464097 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:24.464137 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:24.467496 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:24.471283 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:24.471295 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:24.471325 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:24.471866 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:24.472352 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:24.472357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:24.472361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:24.472679 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:24.473590 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:24.473598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:24.474019 (+ 421us) client_negotiation.cc:769] Sending connection context
0619 18:53:24.474297 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 18:53:24.474871 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":449,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
I20260619 18:53:24.478843 20186 heartbeater.cc:347] Connected to a master server at 127.15.91.62:36317
I20260619 18:53:24.479355 20186 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:24.480402 20186 heartbeater.cc:511] Master 127.15.91.62:36317 requested a full tablet report, sending...
I20260619 18:53:24.482656 19865 ts_manager.cc:194] Registered new tserver with Master: 8f4b29fcd8664a8ab1aaaadfca3f3927 (127.15.91.2:41469)
I20260619 18:53:24.483764 19865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:40763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:53:24.506065 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36317
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:24.871153 20192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:24.871479 20192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:24.871686 20192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:24.879912 20192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:24.880059 20192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:24.880292 20192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:53:24.892678 20192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:24.894557 20192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:24.896525 20192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:24.912113 20197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:24.912161 20200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:24.912070 20198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:24.913158 20192 server_base.cc:1034] running on GCE node
I20260619 18:53:24.913834 20192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:24.915020 20192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:24.916287 20192 hybrid_clock.cc:648] HybridClock initialized: now 1781895204916236 us; error 52 us; skew 500 ppm
I20260619 18:53:24.916671 20192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:24 dist-test-slave-dm3f krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895204, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:24.920713 20192 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:53:24.922096 20192 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:24.922256 20192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:24.922617 20192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:24.926019 20192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "2fab8b9580bd41efbd3e82fd5acfb63b"
format_stamp: "Formatted at 2026-06-19 18:53:24 on dist-test-slave-dm3f"
server_key: "72505e344dd6fbfb8683efc7cb4d3dcf"
server_key_iv: "b7bdde1a3e42443ae7272ad26dab79af"
server_key_version: "encryptionkey@0"
I20260619 18:53:24.927119 20192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "2fab8b9580bd41efbd3e82fd5acfb63b"
format_stamp: "Formatted at 2026-06-19 18:53:24 on dist-test-slave-dm3f"
server_key: "72505e344dd6fbfb8683efc7cb4d3dcf"
server_key_iv: "b7bdde1a3e42443ae7272ad26dab79af"
server_key_version: "encryptionkey@0"
I20260619 18:53:24.933244 20192 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:53:24.933521 20053 heartbeater.cc:503] Master 127.15.91.62:36317 was elected leader, sending a full tablet report...
I20260619 18:53:24.938220 20206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:24.940546 20192 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:53:24.941035 20192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "2fab8b9580bd41efbd3e82fd5acfb63b"
format_stamp: "Formatted at 2026-06-19 18:53:24 on dist-test-slave-dm3f"
server_key: "72505e344dd6fbfb8683efc7cb4d3dcf"
server_key_iv: "b7bdde1a3e42443ae7272ad26dab79af"
server_key_version: "encryptionkey@0"
I20260619 18:53:24.941391 20192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:24.960079 20192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:24.965695 20192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:24.966068 20192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:24.967345 20192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:24.969165 20192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:24.969321 20192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:24.969453 20192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:24.969555 20192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:25.011929 20192 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:42015
I20260619 18:53:25.012008 20318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:42015 every 8 connection(s)
I20260619 18:53:25.013026 20192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:53:25.019409 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 20192
I20260619 18:53:25.019724 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:53:25.020366 15724 external_mini_cluster.cc:1428] Setting key 587a741e67fcd1d1aca9c5ede16717e5
Jun 19 18:53:25 dist-test-slave-dm3f krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895204, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:25.039708 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.016031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:59515 (local address 127.15.91.62:36317)
0619 18:53:25.016573 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:25.016577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:25.017575 (+ 998us) server_negotiation.cc:407] Connection header received
0619 18:53:25.019842 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:25.019849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:25.019988 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:25.020131 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:25.022519 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.023531 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.025546 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.026059 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.026191 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.030522 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:25.030529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:25.030533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:25.030577 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:25.034188 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:25.035591 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:25.035600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:25.035606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:25.035855 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:25.037066 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:25.037073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:25.037076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:25.037764 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:25.037935 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:25.039112 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0619 18:53:25.039374 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":288,"threads_started":1}
I20260619 18:53:25.040771 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.016525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:36317 (local address 127.15.91.3:59515)
0619 18:53:25.017362 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:25.017409 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:25.019366 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:25.020399 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:25.020409 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:25.021303 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:25.022239 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.022275 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.023751 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.023759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.025303 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.025324 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.026324 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.026330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.026575 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.027475 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:25.027515 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:25.030162 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:25.034547 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:25.034558 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:25.034576 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:25.035425 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:25.036161 (+ 736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:25.036172 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:25.036178 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:25.036804 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:25.037958 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:25.037966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:25.038476 (+ 510us) client_negotiation.cc:769] Sending connection context
0619 18:53:25.038887 (+ 411us) client_negotiation.cc:240] Negotiation successful
0619 18:53:25.039395 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":335,"threads_started":1}
I20260619 18:53:25.042694 20319 heartbeater.cc:347] Connected to a master server at 127.15.91.62:36317
I20260619 18:53:25.043170 20319 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:25.044015 20319 heartbeater.cc:511] Master 127.15.91.62:36317 requested a full tablet report, sending...
I20260619 18:53:25.045928 19865 ts_manager.cc:194] Registered new tserver with Master: 2fab8b9580bd41efbd3e82fd5acfb63b (127.15.91.3:42015)
I20260619 18:53:25.047000 19865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:59515
I20260619 18:53:25.050664 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:53:25.073087 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.056596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.15.91.62:36317)
0619 18:53:25.056892 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:25.056898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:25.057368 (+ 470us) server_negotiation.cc:407] Connection header received
0619 18:53:25.057515 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:25.057529 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:25.057625 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:25.057735 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:25.060321 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.061207 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.062823 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.063368 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.063542 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.066290 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:25.066296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:25.066300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:25.066347 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:25.069383 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:25.070367 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:25.070374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:25.070377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:25.070663 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:25.071394 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:25.071401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:25.071403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:25.071861 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:25.072106 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:25.072597 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 18:53:25.072819 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260619 18:53:25.079038 19865 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:53:25.087726 19865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:25.128530 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.101813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42385 (local address 127.0.0.1:54058)
0619 18:53:25.116649 (+ 14836us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:25.116714 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:25.116998 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:25.119355 (+ 2357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:25.119369 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:25.119421 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:25.120587 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.120600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.122414 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.122419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.125416 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.125436 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.127289 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.127296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.127547 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.127613 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 18:53:25.127780 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 18:53:25.128029 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14433,"mutex_wait_us":36,"thread_start_us":17875,"threads_started":1}
I20260619 18:53:25.128427 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.102148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.15.91.1:42385)
0619 18:53:25.118385 (+ 16237us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:25.118399 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:25.118453 (+ 54us) server_negotiation.cc:407] Connection header received
0619 18:53:25.118579 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:25.118588 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:25.118978 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:25.119235 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:25.120788 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.122251 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.125667 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.127063 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.127235 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.127506 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:25.127834 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 18:53:25.128037 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16032,"thread_start_us":15846,"threads_started":1}
I20260619 18:53:25.129455 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.102690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:41469 (local address 127.0.0.1:54496)
0619 18:53:25.120016 (+ 17326us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:25.120038 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:25.120168 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:25.121337 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:25.121344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:25.121370 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:25.121799 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.121813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.123454 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.123460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.125950 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.125962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.128307 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.128317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.128625 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.128717 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 18:53:25.128827 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 18:53:25.128987 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17202,"mutex_wait_us":177,"thread_start_us":16161,"threads_started":1}
I20260619 18:53:25.129508 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.102872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.15.91.3:42015)
0619 18:53:25.112514 (+ 9642us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:25.112527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:25.119365 (+ 6838us) server_negotiation.cc:407] Connection header received
0619 18:53:25.119839 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:25.119850 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:25.120148 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:25.120349 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:25.121553 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.123084 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.126345 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.127906 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.128041 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.128247 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:25.128935 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 18:53:25.129139 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9395,"thread_start_us":9191,"threads_started":1}
I20260619 18:53:25.130461 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.102261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:42015 (local address 127.0.0.1:54920)
0619 18:53:25.119213 (+ 16952us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:25.119235 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:25.119595 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:25.120354 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:25.120358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:25.120880 (+ 522us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:25.121279 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.121293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.123265 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.123273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.126128 (+ 2855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.126144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.128398 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.128406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.128718 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.128745 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:53:25.129848 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0619 18:53:25.130001 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16779,"mutex_wait_us":320,"thread_start_us":13812,"threads_started":1}
I20260619 18:53:25.130605 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.102983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.15.91.2:41469)
0619 18:53:25.118663 (+ 15680us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:25.118671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:25.120444 (+ 1773us) server_negotiation.cc:407] Connection header received
0619 18:53:25.120587 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:25.120595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:25.120865 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:25.121094 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:25.121993 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.123286 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.126151 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.127751 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.127904 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.128148 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:25.129779 (+ 1631us) server_negotiation.cc:299] Negotiation successful
0619 18:53:25.129987 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15525,"thread_start_us":15421,"threads_started":1}
I20260619 18:53:25.134133 20254 tablet_service.cc:1467] Processing CreateTablet for tablet dbe488647918474e9557334504753e3c (DEFAULT_TABLE table=test-table [id=6a2b9e06539c4ec79bf95c18a4f4b555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:25.134263 20121 tablet_service.cc:1467] Processing CreateTablet for tablet dbe488647918474e9557334504753e3c (DEFAULT_TABLE table=test-table [id=6a2b9e06539c4ec79bf95c18a4f4b555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:25.134188 19988 tablet_service.cc:1467] Processing CreateTablet for tablet dbe488647918474e9557334504753e3c (DEFAULT_TABLE table=test-table [id=6a2b9e06539c4ec79bf95c18a4f4b555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:25.135931 20254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe488647918474e9557334504753e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:25.135943 19988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe488647918474e9557334504753e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:25.135941 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe488647918474e9557334504753e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:25.150771 20338 tablet_bootstrap.cc:492] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: Bootstrap starting.
I20260619 18:53:25.154655 20338 tablet_bootstrap.cc:654] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:25.155745 20340 tablet_bootstrap.cc:492] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: Bootstrap starting.
I20260619 18:53:25.156436 20338 log.cc:824] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:25.158258 20339 tablet_bootstrap.cc:492] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: Bootstrap starting.
I20260619 18:53:25.160190 20340 tablet_bootstrap.cc:654] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:25.161661 20340 log.cc:824] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:25.161994 20339 tablet_bootstrap.cc:654] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:25.164098 20339 log.cc:824] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:25.175580 20338 tablet_bootstrap.cc:492] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: No bootstrap required, opened a new log
I20260619 18:53:25.176052 20338 ts_tablet_manager.cc:1397] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.008s
I20260619 18:53:25.182036 20340 tablet_bootstrap.cc:492] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: No bootstrap required, opened a new log
I20260619 18:53:25.182590 20340 ts_tablet_manager.cc:1397] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.022s
I20260619 18:53:25.182688 20339 tablet_bootstrap.cc:492] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: No bootstrap required, opened a new log
I20260619 18:53:25.183043 20339 ts_tablet_manager.cc:1397] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.013s
I20260619 18:53:25.190101 20338 raft_consensus.cc:348] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e8289b6b0e4c2f8db033600b080198" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42385 } } peers { permanent_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 42015 } } peers { permanent_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 41469 } }
I20260619 18:53:25.190596 20338 raft_consensus.cc:374] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:25.190730 20338 raft_consensus.cc:729] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fab8b9580bd41efbd3e82fd5acfb63b, State: Initialized, Role: FOLLOWER
I20260619 18:53:25.191491 20338 consensus_queue.cc:260] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e8289b6b0e4c2f8db033600b080198" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42385 } } peers { permanent_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 42015 } } peers { permanent_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 41469 } }
I20260619 18:53:25.192809 20319 heartbeater.cc:503] Master 127.15.91.62:36317 was elected leader, sending a full tablet report...
I20260619 18:53:25.193537 20338 ts_tablet_manager.cc:1428] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260619 18:53:25.194751 20339 raft_consensus.cc:348] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e8289b6b0e4c2f8db033600b080198" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42385 } } peers { permanent_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 42015 } } peers { permanent_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 41469 } }
I20260619 18:53:25.194746 20340 raft_consensus.cc:348] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e8289b6b0e4c2f8db033600b080198" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42385 } } peers { permanent_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 42015 } } peers { permanent_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 41469 } }
I20260619 18:53:25.195287 20339 raft_consensus.cc:374] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:25.195287 20340 raft_consensus.cc:374] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:25.195400 20340 raft_consensus.cc:729] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4b29fcd8664a8ab1aaaadfca3f3927, State: Initialized, Role: FOLLOWER
I20260619 18:53:25.195398 20339 raft_consensus.cc:729] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e8289b6b0e4c2f8db033600b080198, State: Initialized, Role: FOLLOWER
I20260619 18:53:25.196132 20340 consensus_queue.cc:260] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e8289b6b0e4c2f8db033600b080198" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42385 } } peers { permanent_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 42015 } } peers { permanent_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 41469 } }
I20260619 18:53:25.196132 20339 consensus_queue.cc:260] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e8289b6b0e4c2f8db033600b080198" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42385 } } peers { permanent_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 42015 } } peers { permanent_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 41469 } }
I20260619 18:53:25.198616 20340 ts_tablet_manager.cc:1428] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260619 18:53:25.199560 20339 ts_tablet_manager.cc:1428] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260619 18:53:25.200183 20186 heartbeater.cc:503] Master 127.15.91.62:36317 was elected leader, sending a full tablet report...
W20260619 18:53:25.201220 20187 tablet.cc:2369] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:53:25.264864 20320 tablet.cc:2369] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:53:25.393213 20345 raft_consensus.cc:482] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:25.393620 20345 raft_consensus.cc:504] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e8289b6b0e4c2f8db033600b080198" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42385 } } peers { permanent_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 42015 } } peers { permanent_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 41469 } }
W20260619 18:53:25.394637 20054 tablet.cc:2369] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:53:25.396034 20345 leader_election.cc:288] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fab8b9580bd41efbd3e82fd5acfb63b (127.15.91.3:42015), 8f4b29fcd8664a8ab1aaaadfca3f3927 (127.15.91.2:41469)
I20260619 18:53:25.404990 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.396680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:55697 (local address 127.15.91.3:42015)
0619 18:53:25.396976 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:25.396984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:25.398178 (+ 1194us) server_negotiation.cc:407] Connection header received
0619 18:53:25.398425 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:25.398434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:25.398542 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:25.398864 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:25.399808 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.400601 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.402929 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.403827 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.404052 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.404181 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:25.404557 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 18:53:25.404706 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 18:53:25.405153 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.396961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:45815 (local address 127.15.91.2:41469)
0619 18:53:25.397247 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:25.397257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:25.398491 (+ 1234us) server_negotiation.cc:407] Connection header received
0619 18:53:25.398858 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:25.398863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:25.398934 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:25.399017 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:25.399774 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.400558 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.402941 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.403826 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.404036 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.404123 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:25.404710 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 18:53:25.404881 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 18:53:25.405175 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.396236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:42015 (local address 127.15.91.1:55697)
0619 18:53:25.397993 (+ 1757us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:25.398020 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:25.398237 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:25.399146 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:25.399149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:25.399172 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:25.399617 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.399628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.400774 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.400780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.402625 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.402648 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.404127 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.404134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.404360 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.404392 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:53:25.404524 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 18:53:25.404684 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1539,"mutex_wait_us":47,"thread_start_us":1689,"threads_started":1}
I20260619 18:53:25.405264 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.396834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:41469 (local address 127.15.91.1:45815)
0619 18:53:25.398361 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:25.398382 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:25.398556 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:25.399093 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:25.399098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:25.399136 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:25.399555 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.399571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.400735 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.400743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.402624 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:25.402648 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.404324 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.404329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:25.404470 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.404495 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:53:25.404678 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 18:53:25.404825 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"mutex_wait_us":68,"thread_start_us":619,"threads_started":1}
I20260619 18:53:25.406503 20274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe488647918474e9557334504753e3c" candidate_uuid: "01e8289b6b0e4c2f8db033600b080198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" is_pre_election: true
I20260619 18:53:25.406467 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe488647918474e9557334504753e3c" candidate_uuid: "01e8289b6b0e4c2f8db033600b080198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" is_pre_election: true
I20260619 18:53:25.407155 20274 raft_consensus.cc:2454] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e8289b6b0e4c2f8db033600b080198 in term 0.
I20260619 18:53:25.407155 20141 raft_consensus.cc:2454] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e8289b6b0e4c2f8db033600b080198 in term 0.
I20260619 18:53:25.407948 19943 leader_election.cc:302] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e8289b6b0e4c2f8db033600b080198, 2fab8b9580bd41efbd3e82fd5acfb63b; no voters:
I20260619 18:53:25.408463 20345 raft_consensus.cc:2781] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:25.408628 20345 raft_consensus.cc:482] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:25.408767 20345 raft_consensus.cc:3037] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:25.411293 20345 raft_consensus.cc:504] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e8289b6b0e4c2f8db033600b080198" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42385 } } peers { permanent_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 42015 } } peers { permanent_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 41469 } }
I20260619 18:53:25.412042 20345 leader_election.cc:288] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [CANDIDATE]: Term 1 election: Requested vote from peers 2fab8b9580bd41efbd3e82fd5acfb63b (127.15.91.3:42015), 8f4b29fcd8664a8ab1aaaadfca3f3927 (127.15.91.2:41469)
I20260619 18:53:25.412679 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe488647918474e9557334504753e3c" candidate_uuid: "01e8289b6b0e4c2f8db033600b080198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927"
I20260619 18:53:25.412863 20274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe488647918474e9557334504753e3c" candidate_uuid: "01e8289b6b0e4c2f8db033600b080198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b"
I20260619 18:53:25.412919 20141 raft_consensus.cc:3037] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:25.413127 20274 raft_consensus.cc:3037] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:25.415611 20274 raft_consensus.cc:2454] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e8289b6b0e4c2f8db033600b080198 in term 1.
I20260619 18:53:25.415612 20141 raft_consensus.cc:2454] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e8289b6b0e4c2f8db033600b080198 in term 1.
I20260619 18:53:25.416249 19941 leader_election.cc:302] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e8289b6b0e4c2f8db033600b080198, 8f4b29fcd8664a8ab1aaaadfca3f3927; no voters:
I20260619 18:53:25.416667 20345 raft_consensus.cc:2781] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:25.417140 20345 raft_consensus.cc:686] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 1 LEADER]: Becoming Leader. State: Replica: 01e8289b6b0e4c2f8db033600b080198, State: Running, Role: LEADER
I20260619 18:53:25.417668 20345 consensus_queue.cc:237] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e8289b6b0e4c2f8db033600b080198" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42385 } } peers { permanent_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 42015 } } peers { permanent_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 41469 } }
I20260619 18:53:25.423647 19864 catalog_manager.cc:5495] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01e8289b6b0e4c2f8db033600b080198 (127.15.91.1). New cstate: current_term: 1 leader_uuid: "01e8289b6b0e4c2f8db033600b080198" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e8289b6b0e4c2f8db033600b080198" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 42015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 41469 } health_report { overall_health: UNKNOWN } } }
I20260619 18:53:25.505188 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.498855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.15.91.1:42385)
0619 18:53:25.499076 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:25.499081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:25.499369 (+ 288us) server_negotiation.cc:407] Connection header received
0619 18:53:25.499644 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:25.499648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:25.499732 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:25.499912 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:25.500343 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.501079 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.502601 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.503171 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.503319 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.503886 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:25.504499 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:25.504680 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:25.504806 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 18:53:25.504901 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:53:25.515139 20274 raft_consensus.cc:1261] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b [term 1 FOLLOWER]: Refusing update from remote peer 01e8289b6b0e4c2f8db033600b080198: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:25.515139 20141 raft_consensus.cc:1261] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927 [term 1 FOLLOWER]: Refusing update from remote peer 01e8289b6b0e4c2f8db033600b080198: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:25.516444 20349 consensus_queue.cc:1035] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4b29fcd8664a8ab1aaaadfca3f3927" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 41469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:25.516875 20345 consensus_queue.cc:1035] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fab8b9580bd41efbd3e82fd5acfb63b" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:25.550683 19864 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:25.551076 19864 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:25.554965 19864 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81e93974ad224197966ff741a3bd86c1: Sending DeleteTablet for 3 replicas of tablet dbe488647918474e9557334504753e3c
I20260619 18:53:25.556461 19988 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe488647918474e9557334504753e3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:25 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:54058
I20260619 18:53:25.556461 20254 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe488647918474e9557334504753e3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:25 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:54920
I20260619 18:53:25.556572 20121 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe488647918474e9557334504753e3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:25 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:54496
I20260619 18:53:25.558800 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 19926
I20260619 18:53:25.569203 20363 tablet_replica.cc:330] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: stopping tablet replica
I20260619 18:53:25.570019 20363 raft_consensus.cc:2229] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:25.571323 20363 raft_consensus.cc:2258] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:25.572189 20362 tablet_replica.cc:330] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: stopping tablet replica
I20260619 18:53:25.572831 20362 raft_consensus.cc:2229] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:25.573738 20362 raft_consensus.cc:2258] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:25.574174 20363 ts_tablet_manager.cc:1905] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:25.576346 20362 ts_tablet_manager.cc:1905] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:25.580749 20364 tablet_replica.cc:330] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: stopping tablet replica
I20260619 18:53:25.581419 20364 raft_consensus.cc:2229] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:25.582093 20364 raft_consensus.cc:2258] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:25.582508 20363 ts_tablet_manager.cc:1918] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:25.583120 20363 log.cc:1196] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/dbe488647918474e9557334504753e3c
I20260619 18:53:25.583833 20363 ts_tablet_manager.cc:1939] T dbe488647918474e9557334504753e3c P 01e8289b6b0e4c2f8db033600b080198: Deleting consensus metadata
I20260619 18:53:25.584285 20362 ts_tablet_manager.cc:1918] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:25.584492 20362 log.cc:1196] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/dbe488647918474e9557334504753e3c
I20260619 18:53:25.585029 20362 ts_tablet_manager.cc:1939] T dbe488647918474e9557334504753e3c P 2fab8b9580bd41efbd3e82fd5acfb63b: Deleting consensus metadata
I20260619 18:53:25.586387 19850 catalog_manager.cc:4841] TS 01e8289b6b0e4c2f8db033600b080198 (127.15.91.1:42385): tablet dbe488647918474e9557334504753e3c (table test-table [id=6a2b9e06539c4ec79bf95c18a4f4b555]) successfully deleted
I20260619 18:53:25.586877 19851 catalog_manager.cc:4841] TS 2fab8b9580bd41efbd3e82fd5acfb63b (127.15.91.3:42015): tablet dbe488647918474e9557334504753e3c (table test-table [id=6a2b9e06539c4ec79bf95c18a4f4b555]) successfully deleted
I20260619 18:53:25.587143 20364 ts_tablet_manager.cc:1905] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:53:25 dist-test-slave-dm3f krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:53:25.595170 20364 ts_tablet_manager.cc:1918] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:25.595420 20364 log.cc:1196] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/dbe488647918474e9557334504753e3c
I20260619 18:53:25.596026 20364 ts_tablet_manager.cc:1939] T dbe488647918474e9557334504753e3c P 8f4b29fcd8664a8ab1aaaadfca3f3927: Deleting consensus metadata
I20260619 18:53:25.598299 19849 catalog_manager.cc:4841] TS 8f4b29fcd8664a8ab1aaaadfca3f3927 (127.15.91.2:41469): tablet dbe488647918474e9557334504753e3c (table test-table [id=6a2b9e06539c4ec79bf95c18a4f4b555]) successfully deleted
I20260619 18:53:25.602028 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.560963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.15.91.1:42385)
0619 18:53:25.561226 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:25.561235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:25.571841 (+ 10606us) server_negotiation.cc:407] Connection header received
0619 18:53:25.572542 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:25.572552 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:25.572688 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:25.572787 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:25.575946 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.577599 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.582840 (+ 5241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.583680 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.583820 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.594858 (+ 11038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:25.594874 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:25.594892 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:25.594941 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:25.598705 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:25.599645 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:25.599652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:25.599654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:25.599827 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:25.600263 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:25.600268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:25.600273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:25.600683 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:25.601026 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:25.601367 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 18:53:25.601652 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 18:53:25.603111 20048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:25.706804 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 19926
I20260619 18:53:25.728900 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 20059
Jun 19 18:53:25 dist-test-slave-dm3f krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:53:25.746909 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.729981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.15.91.2:41469)
0619 18:53:25.730216 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:25.730225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:25.730250 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:53:25.730505 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:25.730510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:25.730582 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:25.730729 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:25.732160 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.732911 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.734305 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.734945 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.735104 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.740013 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:25.740033 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:25.740051 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:25.740098 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:25.743426 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:25.744307 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:25.744315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:25.744318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:25.744497 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:25.744936 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:25.744941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:25.744943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:25.745408 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:25.745747 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:25.746161 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 18:53:25.746362 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 18:53:25.747663 20181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:25.844376 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 20059
I20260619 18:53:25.862154 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 20192
Jun 19 18:53:25 dist-test-slave-dm3f krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:53:25.878356 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:25.863025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.15.91.3:42015)
0619 18:53:25.863210 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:25.863215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:25.863292 (+ 77us) server_negotiation.cc:407] Connection header received
0619 18:53:25.863455 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:25.863460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:25.863527 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:25.863626 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:25.864962 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.865702 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.867282 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:25.867707 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:25.867816 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:25.872459 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:25.872476 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:25.872494 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:25.872540 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:25.875484 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:25.876401 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:25.876410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:25.876412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:25.876577 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:25.876971 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:25.876975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:25.876977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:25.877350 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:25.877590 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:25.877872 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 18:53:25.878066 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:53:25.879426 20314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:25.971654 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 20192
I20260619 18:53:25.990664 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 19834
I20260619 18:53:25.992156 19895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:26.103372 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 19834
2026-06-19T18:53:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4755 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T18:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:26Z Disabled control of system clock
I20260619 18:53:26.175523 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:43081
--webserver_interface=127.15.91.62
--webserver_port=0
--builtin_ntp_servers=127.15.91.20:33083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:43081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:53:26.533154 20373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:26.533511 20373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:26.533668 20373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:26.541512 20373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:26.541646 20373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:26.541730 20373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:26.541821 20373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:26.541906 20373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:26.551949 20373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:43081
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:43081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:26.553409 20373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:26.555393 20373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:26.568862 20378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:26.568861 20381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:26.568861 20379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:26.569711 20373 server_base.cc:1034] running on GCE node
I20260619 18:53:26.570667 20373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:26.572434 20373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:26.573727 20373 hybrid_clock.cc:648] HybridClock initialized: now 1781895206573679 us; error 77 us; skew 500 ppm
I20260619 18:53:26.574137 20373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:53:26.576926 20373 webserver.cc:460] Webserver started at http://127.15.91.62:39735/ using document root <none> and password file <none>
I20260619 18:53:26.577783 20373 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:26.577945 20373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:26.578369 20373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:26.581770 20373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "7cc23e38846f49b09cf42732ebf800c0"
format_stamp: "Formatted at 2026-06-19 18:53:26 on dist-test-slave-dm3f"
server_key: "bc8f22c94960deeafb94a4edab044224"
server_key_iv: "86acc6d719e119e252871a0972122042"
server_key_version: "encryptionkey@0"
I20260619 18:53:26.582895 20373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "7cc23e38846f49b09cf42732ebf800c0"
format_stamp: "Formatted at 2026-06-19 18:53:26 on dist-test-slave-dm3f"
server_key: "bc8f22c94960deeafb94a4edab044224"
server_key_iv: "86acc6d719e119e252871a0972122042"
server_key_version: "encryptionkey@0"
I20260619 18:53:26.589246 20373 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:53:26.593551 20387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:26.595348 20373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:53:26.595639 20373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "7cc23e38846f49b09cf42732ebf800c0"
format_stamp: "Formatted at 2026-06-19 18:53:26 on dist-test-slave-dm3f"
server_key: "bc8f22c94960deeafb94a4edab044224"
server_key_iv: "86acc6d719e119e252871a0972122042"
server_key_version: "encryptionkey@0"
I20260619 18:53:26.595942 20373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:26.602756 20373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:26.603761 20373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:26.604115 20373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:26.625047 20373 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:43081
I20260619 18:53:26.625070 20438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:43081 every 8 connection(s)
I20260619 18:53:26.626673 20373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:26.628214 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 20373
I20260619 18:53:26.628422 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:26.629119 15724 external_mini_cluster.cc:1428] Setting key 96a508e3634af4c0d1be8ec7812e680e
I20260619 18:53:26.631910 20439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:26.640376 20442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:26.631486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.15.91.62:43081)
0619 18:53:26.632788 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:26.632805 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:26.632868 (+ 63us) server_negotiation.cc:407] Connection header received
0619 18:53:26.635328 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:26.635364 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:26.636096 (+ 732us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:26.636679 (+ 583us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:26.637886 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:26.637895 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:26.637920 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:26.637942 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:26.638058 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:26.638088 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:26.638353 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:26.638525 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 18:53:26.638908 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":444,"threads_started":1}
I20260619 18:53:26.646922 20439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0: Bootstrap starting.
I20260619 18:53:26.650543 20439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:26.651832 20439 log.cc:824] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:26.656154 20439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0: No bootstrap required, opened a new log
I20260619 18:53:26.665017 20439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc23e38846f49b09cf42732ebf800c0" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43081 } }
I20260619 18:53:26.665481 20439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:26.665591 20439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc23e38846f49b09cf42732ebf800c0, State: Initialized, Role: FOLLOWER
I20260619 18:53:26.666285 20439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc23e38846f49b09cf42732ebf800c0" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43081 } }
I20260619 18:53:26.666587 20439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:26.666769 20439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:26.666987 20439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:26.669348 20439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc23e38846f49b09cf42732ebf800c0" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43081 } }
I20260619 18:53:26.670044 20439 leader_election.cc:302] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cc23e38846f49b09cf42732ebf800c0; no voters:
I20260619 18:53:26.670917 20439 leader_election.cc:288] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:26.671021 20444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:26.671885 20444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [term 1 LEADER]: Becoming Leader. State: Replica: 7cc23e38846f49b09cf42732ebf800c0, State: Running, Role: LEADER
I20260619 18:53:26.672648 20444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc23e38846f49b09cf42732ebf800c0" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43081 } }
I20260619 18:53:26.673714 20439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:26.676304 20445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cc23e38846f49b09cf42732ebf800c0. Latest consensus state: current_term: 1 leader_uuid: "7cc23e38846f49b09cf42732ebf800c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc23e38846f49b09cf42732ebf800c0" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43081 } } }
I20260619 18:53:26.676867 20445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:26.676494 20446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cc23e38846f49b09cf42732ebf800c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc23e38846f49b09cf42732ebf800c0" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43081 } } }
I20260619 18:53:26.677011 20446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:26.682279 20451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:26.688813 20451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:26.700031 20451 catalog_manager.cc:1347] Generated new cluster ID: dd65d884976743b18ffc74d8f598778a
I20260619 18:53:26.700219 20451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:26.714044 20451 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:26.714743 20451 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:26.729516 20451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0: Generated new TSK 0
I20260619 18:53:26.730701 20451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:53:26.744619 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43081
--builtin_ntp_servers=127.15.91.20:33083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:53:27.132485 20463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:27.132814 20463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:27.132972 20463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:27.140985 20463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:27.141113 20463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:27.141309 20463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:53:27.151319 20463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:27.152740 20463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:27.154870 20463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:27.169505 20471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:27.169776 20469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:27.169552 20468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:27.172327 20463 server_base.cc:1034] running on GCE node
I20260619 18:53:27.173099 20463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:27.174218 20463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:27.175458 20463 hybrid_clock.cc:648] HybridClock initialized: now 1781895207175413 us; error 45 us; skew 500 ppm
I20260619 18:53:27.175832 20463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:53:27.178763 20463 webserver.cc:460] Webserver started at http://127.15.91.1:41241/ using document root <none> and password file <none>
I20260619 18:53:27.179737 20463 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:27.179879 20463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:27.180240 20463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:27.183877 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "2bc9d497340e45b19f010308a7b8872f"
format_stamp: "Formatted at 2026-06-19 18:53:27 on dist-test-slave-dm3f"
server_key: "77c3c8e4aeb95cb5dc4f4cc1ffd35a16"
server_key_iv: "9a9fdc094528203ae872e5230b66e902"
server_key_version: "encryptionkey@0"
I20260619 18:53:27.184842 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "2bc9d497340e45b19f010308a7b8872f"
format_stamp: "Formatted at 2026-06-19 18:53:27 on dist-test-slave-dm3f"
server_key: "77c3c8e4aeb95cb5dc4f4cc1ffd35a16"
server_key_iv: "9a9fdc094528203ae872e5230b66e902"
server_key_version: "encryptionkey@0"
I20260619 18:53:27.191205 20463 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:53:27.195412 20477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:27.197111 20463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:53:27.197373 20463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "2bc9d497340e45b19f010308a7b8872f"
format_stamp: "Formatted at 2026-06-19 18:53:27 on dist-test-slave-dm3f"
server_key: "77c3c8e4aeb95cb5dc4f4cc1ffd35a16"
server_key_iv: "9a9fdc094528203ae872e5230b66e902"
server_key_version: "encryptionkey@0"
I20260619 18:53:27.197669 20463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:27.204972 20463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:27.206034 20463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:27.206375 20463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:27.207644 20463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:27.209367 20463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:27.209523 20463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:27.209681 20463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:27.209777 20463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:27.246562 20463 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:34791
I20260619 18:53:27.246608 20589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:34791 every 8 connection(s)
I20260619 18:53:27.248217 20463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:53:27.251439 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 20463
I20260619 18:53:27.251682 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:53:27.252409 15724 external_mini_cluster.cc:1428] Setting key 5de9e2ce8493769ff66566ebd5f9703c
I20260619 18:53:27.255532 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43081
--builtin_ntp_servers=127.15.91.20:33083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 18:53:27.258423 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:27.251317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:37155 (local address 127.15.91.62:43081)
0619 18:53:27.251854 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:27.251862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:27.253637 (+ 1775us) server_negotiation.cc:407] Connection header received
0619 18:53:27.255632 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:27.255638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:27.255734 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:27.255858 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:27.257538 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:27.257544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:27.257549 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:27.257579 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:27.257625 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:27.257638 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:27.257793 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:27.258017 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 18:53:27.258098 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":256,"threads_started":1}
I20260619 18:53:27.259603 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:27.252073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43081 (local address 127.15.91.1:37155)
0619 18:53:27.253448 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:27.253503 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:27.255288 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:27.256180 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:27.256190 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:27.256306 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:27.257138 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:53:27.257140 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:27.257196 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:53:27.257304 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:27.257748 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:27.257761 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:27.257772 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:53:27.258122 (+ 350us) client_negotiation.cc:240] Negotiation successful
0619 18:53:27.258449 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":482,"threads_started":1}
I20260619 18:53:27.261541 20590 heartbeater.cc:347] Connected to a master server at 127.15.91.62:43081
I20260619 18:53:27.261945 20590 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:27.262823 20590 heartbeater.cc:511] Master 127.15.91.62:43081 requested a full tablet report, sending...
I20260619 18:53:27.265524 20404 ts_manager.cc:194] Registered new tserver with Master: 2bc9d497340e45b19f010308a7b8872f (127.15.91.1:34791)
W20260619 18:53:27.615093 20594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:27.615417 20594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:27.615553 20594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:27.623515 20594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:27.623646 20594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:27.623835 20594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:53:27.639375 20594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:27.641042 20594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:27.643613 20594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:27.659679 20600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:27.659654 20602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:27.659945 20599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:27.661908 20594 server_base.cc:1034] running on GCE node
I20260619 18:53:27.662616 20594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:27.663808 20594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:27.665078 20594 hybrid_clock.cc:648] HybridClock initialized: now 1781895207665014 us; error 57 us; skew 500 ppm
I20260619 18:53:27.665477 20594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:53:27.668239 20594 webserver.cc:460] Webserver started at http://127.15.91.2:33923/ using document root <none> and password file <none>
I20260619 18:53:27.669106 20594 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:27.669248 20594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:27.669638 20594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:27.673018 20594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "7113963474f748cdbf0af9f36ca536d7"
format_stamp: "Formatted at 2026-06-19 18:53:27 on dist-test-slave-dm3f"
server_key: "3f55675e15aa718a077e6a95cc9f452a"
server_key_iv: "118d8189d878158df899d4539c884d7b"
server_key_version: "encryptionkey@0"
I20260619 18:53:27.674069 20594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "7113963474f748cdbf0af9f36ca536d7"
format_stamp: "Formatted at 2026-06-19 18:53:27 on dist-test-slave-dm3f"
server_key: "3f55675e15aa718a077e6a95cc9f452a"
server_key_iv: "118d8189d878158df899d4539c884d7b"
server_key_version: "encryptionkey@0"
I20260619 18:53:27.680275 20594 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:53:27.684640 20608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:27.686311 20594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:53:27.686563 20594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "7113963474f748cdbf0af9f36ca536d7"
format_stamp: "Formatted at 2026-06-19 18:53:27 on dist-test-slave-dm3f"
server_key: "3f55675e15aa718a077e6a95cc9f452a"
server_key_iv: "118d8189d878158df899d4539c884d7b"
server_key_version: "encryptionkey@0"
I20260619 18:53:27.686913 20594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:27.693467 20594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:27.694500 20594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:27.694902 20594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:27.695973 20594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:27.697640 20594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:27.697785 20594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:27.697932 20594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:27.698022 20594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:27.735167 20594 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:37567
I20260619 18:53:27.735200 20720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:37567 every 8 connection(s)
I20260619 18:53:27.736649 20594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:53:27.739425 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 20594
I20260619 18:53:27.739625 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:53:27.740170 15724 external_mini_cluster.cc:1428] Setting key 157f4d743f805ba02d5440bfe6b56f00
I20260619 18:53:27.743759 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43081
--builtin_ntp_servers=127.15.91.20:33083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 18:53:27.747821 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:27.739592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:60399 (local address 127.15.91.62:43081)
0619 18:53:27.739831 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:27.739838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:27.741705 (+ 1867us) server_negotiation.cc:407] Connection header received
0619 18:53:27.744413 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:27.744418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:27.744496 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:27.744599 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:27.746860 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:27.746869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:27.746877 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:27.746905 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:27.746954 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:27.746973 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:27.747156 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:27.747424 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 18:53:27.747496 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:53:27.749315 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:27.740209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43081 (local address 127.15.91.2:60399)
0619 18:53:27.741540 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:27.741582 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:27.743958 (+ 2376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:27.744955 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:27.744972 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:27.745138 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:27.746163 (+ 1025us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:53:27.746169 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:27.746234 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:53:27.746383 (+ 149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:27.747131 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:27.747148 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:27.747161 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 18:53:27.747522 (+ 361us) client_negotiation.cc:240] Negotiation successful
0619 18:53:27.747854 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":390,"threads_started":1}
I20260619 18:53:27.751171 20721 heartbeater.cc:347] Connected to a master server at 127.15.91.62:43081
I20260619 18:53:27.751536 20721 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:27.752220 20721 heartbeater.cc:511] Master 127.15.91.62:43081 requested a full tablet report, sending...
I20260619 18:53:27.754076 20404 ts_manager.cc:194] Registered new tserver with Master: 7113963474f748cdbf0af9f36ca536d7 (127.15.91.2:37567)
W20260619 18:53:28.103250 20724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:28.103579 20724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:28.103674 20724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:28.111567 20724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:28.111694 20724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:28.111889 20724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:53:28.121724 20724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:28.123154 20724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:28.125104 20724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:28.139308 20730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:28.139257 20729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:28.140453 20732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:28.141655 20724 server_base.cc:1034] running on GCE node
I20260619 18:53:28.142418 20724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:28.143668 20724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:28.144910 20724 hybrid_clock.cc:648] HybridClock initialized: now 1781895208144852 us; error 48 us; skew 500 ppm
I20260619 18:53:28.145303 20724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:53:28.148181 20724 webserver.cc:460] Webserver started at http://127.15.91.3:35155/ using document root <none> and password file <none>
I20260619 18:53:28.149016 20724 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:28.149161 20724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:28.149547 20724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:28.152894 20724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "4628229a7d9c449b9200000e808be653"
format_stamp: "Formatted at 2026-06-19 18:53:28 on dist-test-slave-dm3f"
server_key: "f9d24a48525d71221aa03abafd1da4cd"
server_key_iv: "a3128cd4b47b24a2b2b2f039a4880a07"
server_key_version: "encryptionkey@0"
I20260619 18:53:28.153879 20724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "4628229a7d9c449b9200000e808be653"
format_stamp: "Formatted at 2026-06-19 18:53:28 on dist-test-slave-dm3f"
server_key: "f9d24a48525d71221aa03abafd1da4cd"
server_key_iv: "a3128cd4b47b24a2b2b2f039a4880a07"
server_key_version: "encryptionkey@0"
I20260619 18:53:28.160075 20724 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:53:28.164382 20738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:28.165997 20724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:53:28.166219 20724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "4628229a7d9c449b9200000e808be653"
format_stamp: "Formatted at 2026-06-19 18:53:28 on dist-test-slave-dm3f"
server_key: "f9d24a48525d71221aa03abafd1da4cd"
server_key_iv: "a3128cd4b47b24a2b2b2f039a4880a07"
server_key_version: "encryptionkey@0"
I20260619 18:53:28.166499 20724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:28.173043 20724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:28.174090 20724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:28.174407 20724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:28.175618 20724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:28.177259 20724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:28.177398 20724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:28.177541 20724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:28.177634 20724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:28.214517 20724 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:44849
I20260619 18:53:28.214574 20850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:44849 every 8 connection(s)
I20260619 18:53:28.216091 20724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:53:28.218449 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 20724
I20260619 18:53:28.218654 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:53:28.219231 15724 external_mini_cluster.cc:1428] Setting key d3f8606278775b08308a1090d7378ee7
I20260619 18:53:28.227308 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.219001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:46553 (local address 127.15.91.62:43081)
0619 18:53:28.219226 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:28.219233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:28.221898 (+ 2665us) server_negotiation.cc:407] Connection header received
0619 18:53:28.224344 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:28.224349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:28.224411 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:28.224531 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:28.226423 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:28.226428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:28.226431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:28.226467 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:28.226505 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:28.226518 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:28.226640 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:28.226983 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 18:53:28.227049 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 18:53:28.228432 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.219582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43081 (local address 127.15.91.3:46553)
0619 18:53:28.221490 (+ 1908us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:28.221673 (+ 183us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:28.223991 (+ 2318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:28.224882 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:28.224893 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:28.225025 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:28.225995 (+ 970us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:53:28.225998 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:28.226048 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:53:28.226166 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:28.226636 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:28.226645 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:28.226651 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:53:28.227022 (+ 371us) client_negotiation.cc:240] Negotiation successful
0619 18:53:28.227292 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":603,"threads_started":1}
I20260619 18:53:28.230253 20851 heartbeater.cc:347] Connected to a master server at 127.15.91.62:43081
I20260619 18:53:28.230664 20851 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:28.231436 20851 heartbeater.cc:511] Master 127.15.91.62:43081 requested a full tablet report, sending...
I20260619 18:53:28.233774 20404 ts_manager.cc:194] Registered new tserver with Master: 4628229a7d9c449b9200000e808be653 (127.15.91.3:44849)
I20260619 18:53:28.237277 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:53:28.245494 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.242404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.15.91.62:43081)
0619 18:53:28.242794 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:28.242804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:28.243270 (+ 466us) server_negotiation.cc:407] Connection header received
0619 18:53:28.243524 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:28.243529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:28.243594 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:28.243711 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:28.244776 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:28.244783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:28.244788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:28.244818 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:28.244854 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:28.244873 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:28.245025 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:28.245122 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 18:53:28.245193 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:53:28.250761 20404 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:53:28.258499 20404 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:28.269569 20590 heartbeater.cc:503] Master 127.15.91.62:43081 was elected leader, sending a full tablet report...
I20260619 18:53:28.281514 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.273336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:37567 (local address 127.0.0.1:59994)
0619 18:53:28.276024 (+ 2688us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:28.276045 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:28.276180 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:28.278245 (+ 2065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:28.278251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:28.278339 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:28.279895 (+ 1556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:53:28.279901 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:28.279937 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:53:28.280053 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:28.280658 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:28.280673 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:28.280681 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:53:28.280877 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 18:53:28.280989 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2527,"thread_start_us":2364,"threads_started":1}
I20260619 18:53:28.281637 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.275331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:44849 (local address 127.0.0.1:49120)
0619 18:53:28.277075 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:28.277103 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:28.277244 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:28.279238 (+ 1994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:28.279246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:28.279280 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:28.280325 (+ 1045us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:53:28.280328 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:28.280345 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:53:28.280385 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:28.280948 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:28.280952 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:28.280956 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:53:28.281124 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 18:53:28.281196 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1564,"thread_start_us":1370,"threads_started":1}
I20260619 18:53:28.282429 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.273798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59994 (local address 127.15.91.2:37567)
0619 18:53:28.275526 (+ 1728us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:28.275538 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:28.276359 (+ 821us) server_negotiation.cc:407] Connection header received
0619 18:53:28.276474 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:28.276487 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:28.276810 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:28.277032 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:28.280278 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:28.280291 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:28.280308 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:28.280331 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:28.280409 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:28.280438 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:28.280602 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:28.281946 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0619 18:53:28.282079 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1537,"thread_start_us":1398,"threads_started":1}
I20260619 18:53:28.282673 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.276016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.15.91.3:44849)
0619 18:53:28.278019 (+ 2003us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:28.278034 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:28.278084 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:53:28.278188 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:28.278202 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:28.278581 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:28.278845 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:28.280584 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:28.280593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:28.280603 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:28.280627 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:28.280685 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:28.280706 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:28.280881 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:28.282184 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0619 18:53:28.282283 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1760,"thread_start_us":1536,"threads_started":1}
I20260619 18:53:28.282909 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.273055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:34791 (local address 127.0.0.1:45428)
0619 18:53:28.274083 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:28.274130 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:28.274280 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:28.279833 (+ 5553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:28.279839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:28.279880 (+ 41us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:28.280552 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:53:28.280554 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:28.280572 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:53:28.280612 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:28.282069 (+ 1457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:28.282076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:28.282083 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:53:28.282248 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 18:53:28.282302 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":580,"threads_started":1}
I20260619 18:53:28.282886 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.273320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.15.91.1:34791)
0619 18:53:28.279247 (+ 5927us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:28.279256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:28.279287 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:53:28.279360 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:28.279369 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:28.279623 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:28.280348 (+ 725us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:28.281768 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:28.281778 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:28.281792 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:28.281809 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:28.281869 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:28.281886 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:28.282158 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:28.282234 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 18:53:28.282334 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5739,"thread_start_us":5585,"threads_started":1}
I20260619 18:53:28.287137 20656 tablet_service.cc:1467] Processing CreateTablet for tablet 931a07f9d7594f28913fe6e2e07f3d1f (DEFAULT_TABLE table=test-table [id=999cbd4b6098460e8a30c323f7598c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:28.287137 20525 tablet_service.cc:1467] Processing CreateTablet for tablet 931a07f9d7594f28913fe6e2e07f3d1f (DEFAULT_TABLE table=test-table [id=999cbd4b6098460e8a30c323f7598c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:28.288749 20656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931a07f9d7594f28913fe6e2e07f3d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:28.288745 20525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931a07f9d7594f28913fe6e2e07f3d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:28.291399 20786 tablet_service.cc:1467] Processing CreateTablet for tablet 931a07f9d7594f28913fe6e2e07f3d1f (DEFAULT_TABLE table=test-table [id=999cbd4b6098460e8a30c323f7598c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:28.293268 20786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931a07f9d7594f28913fe6e2e07f3d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:28.303261 20869 tablet_bootstrap.cc:492] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f: Bootstrap starting.
I20260619 18:53:28.305783 20870 tablet_bootstrap.cc:492] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: Bootstrap starting.
I20260619 18:53:28.307888 20869 tablet_bootstrap.cc:654] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:28.309849 20869 log.cc:824] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:28.310496 20870 tablet_bootstrap.cc:654] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:28.310751 20871 tablet_bootstrap.cc:492] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: Bootstrap starting.
I20260619 18:53:28.312974 20870 log.cc:824] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:28.314848 20869 tablet_bootstrap.cc:492] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f: No bootstrap required, opened a new log
I20260619 18:53:28.314991 20871 tablet_bootstrap.cc:654] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:28.315220 20869 ts_tablet_manager.cc:1397] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260619 18:53:28.316514 20871 log.cc:824] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:28.316921 20870 tablet_bootstrap.cc:492] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: No bootstrap required, opened a new log
I20260619 18:53:28.317289 20870 ts_tablet_manager.cc:1397] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260619 18:53:28.320580 20871 tablet_bootstrap.cc:492] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: No bootstrap required, opened a new log
I20260619 18:53:28.320983 20871 ts_tablet_manager.cc:1397] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260619 18:53:28.325794 20870 raft_consensus.cc:348] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc9d497340e45b19f010308a7b8872f" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 34791 } } peers { permanent_uuid: "7113963474f748cdbf0af9f36ca536d7" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 37567 } } peers { permanent_uuid: "4628229a7d9c449b9200000e808be653" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44849 } }
I20260619 18:53:28.326306 20870 raft_consensus.cc:374] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:28.326433 20870 raft_consensus.cc:729] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7113963474f748cdbf0af9f36ca536d7, State: Initialized, Role: FOLLOWER
I20260619 18:53:28.326462 20869 raft_consensus.cc:348] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc9d497340e45b19f010308a7b8872f" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 34791 } } peers { permanent_uuid: "7113963474f748cdbf0af9f36ca536d7" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 37567 } } peers { permanent_uuid: "4628229a7d9c449b9200000e808be653" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44849 } }
I20260619 18:53:28.327008 20869 raft_consensus.cc:374] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:28.327131 20869 raft_consensus.cc:729] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc9d497340e45b19f010308a7b8872f, State: Initialized, Role: FOLLOWER
I20260619 18:53:28.327282 20870 consensus_queue.cc:260] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc9d497340e45b19f010308a7b8872f" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 34791 } } peers { permanent_uuid: "7113963474f748cdbf0af9f36ca536d7" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 37567 } } peers { permanent_uuid: "4628229a7d9c449b9200000e808be653" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44849 } }
I20260619 18:53:28.327808 20869 consensus_queue.cc:260] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc9d497340e45b19f010308a7b8872f" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 34791 } } peers { permanent_uuid: "7113963474f748cdbf0af9f36ca536d7" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 37567 } } peers { permanent_uuid: "4628229a7d9c449b9200000e808be653" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44849 } }
I20260619 18:53:28.328650 20721 heartbeater.cc:503] Master 127.15.91.62:43081 was elected leader, sending a full tablet report...
I20260619 18:53:28.329424 20870 ts_tablet_manager.cc:1428] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260619 18:53:28.334456 20869 ts_tablet_manager.cc:1428] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20260619 18:53:28.334591 20871 raft_consensus.cc:348] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc9d497340e45b19f010308a7b8872f" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 34791 } } peers { permanent_uuid: "7113963474f748cdbf0af9f36ca536d7" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 37567 } } peers { permanent_uuid: "4628229a7d9c449b9200000e808be653" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44849 } }
I20260619 18:53:28.335233 20871 raft_consensus.cc:374] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:28.335353 20871 raft_consensus.cc:729] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4628229a7d9c449b9200000e808be653, State: Initialized, Role: FOLLOWER
I20260619 18:53:28.336154 20871 consensus_queue.cc:260] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc9d497340e45b19f010308a7b8872f" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 34791 } } peers { permanent_uuid: "7113963474f748cdbf0af9f36ca536d7" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 37567 } } peers { permanent_uuid: "4628229a7d9c449b9200000e808be653" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44849 } }
I20260619 18:53:28.337464 20851 heartbeater.cc:503] Master 127.15.91.62:43081 was elected leader, sending a full tablet report...
I20260619 18:53:28.338271 20871 ts_tablet_manager.cc:1428] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260619 18:53:28.467976 20852 tablet.cc:2369] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:53:28.489131 20722 tablet.cc:2369] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:53:28.500921 20591 tablet.cc:2369] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:53:28.527346 20875 raft_consensus.cc:482] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:28.527729 20875 raft_consensus.cc:504] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc9d497340e45b19f010308a7b8872f" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 34791 } } peers { permanent_uuid: "7113963474f748cdbf0af9f36ca536d7" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 37567 } } peers { permanent_uuid: "4628229a7d9c449b9200000e808be653" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44849 } }
I20260619 18:53:28.529673 20875 leader_election.cc:288] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bc9d497340e45b19f010308a7b8872f (127.15.91.1:34791), 4628229a7d9c449b9200000e808be653 (127.15.91.3:44849)
I20260619 18:53:28.533711 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.530290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:34695 (local address 127.15.91.3:44849)
0619 18:53:28.530550 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:28.530571 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:28.531348 (+ 777us) server_negotiation.cc:407] Connection header received
0619 18:53:28.531540 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:28.531545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:28.531599 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:28.531724 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:28.533014 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:28.533020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:28.533025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:28.533061 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:28.533097 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:28.533110 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:28.533268 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:28.533352 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 18:53:28.533425 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:53:28.533708 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.530151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:43691 (local address 127.15.91.1:34791)
0619 18:53:28.530408 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:28.530419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:28.531302 (+ 883us) server_negotiation.cc:407] Connection header received
0619 18:53:28.531544 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:28.531549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:28.531601 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:28.531722 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:28.533054 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:28.533059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:28.533062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:28.533081 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:28.533115 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:28.533127 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:28.533278 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:28.533339 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 18:53:28.533416 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:53:28.534044 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.529902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:34791 (local address 127.15.91.2:43691)
0619 18:53:28.531148 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:28.531174 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:28.531365 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:28.531756 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:28.531761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:28.531789 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:28.532806 (+ 1017us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:53:28.532808 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:28.532823 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:53:28.532865 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:28.533234 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:28.533239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:28.533243 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:53:28.533421 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 18:53:28.533499 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"thread_start_us":960,"threads_started":1}
I20260619 18:53:28.534108 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.530086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:44849 (local address 127.15.91.2:34695)
0619 18:53:28.531211 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:28.531229 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:28.531390 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:28.531756 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:28.531763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:28.531790 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:28.532743 (+ 953us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:53:28.532746 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:28.532772 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:53:28.532822 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:28.533222 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:28.533227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:28.533232 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:53:28.533446 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 18:53:28.533519 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":3,"thread_start_us":624,"threads_started":1}
I20260619 18:53:28.535490 20806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931a07f9d7594f28913fe6e2e07f3d1f" candidate_uuid: "7113963474f748cdbf0af9f36ca536d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4628229a7d9c449b9200000e808be653" is_pre_election: true
I20260619 18:53:28.535459 20545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931a07f9d7594f28913fe6e2e07f3d1f" candidate_uuid: "7113963474f748cdbf0af9f36ca536d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc9d497340e45b19f010308a7b8872f" is_pre_election: true
I20260619 18:53:28.536052 20545 raft_consensus.cc:2454] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7113963474f748cdbf0af9f36ca536d7 in term 0.
I20260619 18:53:28.536062 20806 raft_consensus.cc:2454] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7113963474f748cdbf0af9f36ca536d7 in term 0.
I20260619 18:53:28.536969 20611 leader_election.cc:302] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2bc9d497340e45b19f010308a7b8872f, 4628229a7d9c449b9200000e808be653, 7113963474f748cdbf0af9f36ca536d7; no voters:
I20260619 18:53:28.537555 20875 raft_consensus.cc:2781] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:28.537742 20875 raft_consensus.cc:482] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:28.537859 20875 raft_consensus.cc:3037] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:28.540380 20875 raft_consensus.cc:504] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc9d497340e45b19f010308a7b8872f" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 34791 } } peers { permanent_uuid: "7113963474f748cdbf0af9f36ca536d7" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 37567 } } peers { permanent_uuid: "4628229a7d9c449b9200000e808be653" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44849 } }
I20260619 18:53:28.541169 20875 leader_election.cc:288] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [CANDIDATE]: Term 1 election: Requested vote from peers 2bc9d497340e45b19f010308a7b8872f (127.15.91.1:34791), 4628229a7d9c449b9200000e808be653 (127.15.91.3:44849)
I20260619 18:53:28.541798 20545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931a07f9d7594f28913fe6e2e07f3d1f" candidate_uuid: "7113963474f748cdbf0af9f36ca536d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc9d497340e45b19f010308a7b8872f"
I20260619 18:53:28.542073 20545 raft_consensus.cc:3037] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:28.541980 20806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931a07f9d7594f28913fe6e2e07f3d1f" candidate_uuid: "7113963474f748cdbf0af9f36ca536d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4628229a7d9c449b9200000e808be653"
I20260619 18:53:28.542501 20806 raft_consensus.cc:3037] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:28.544716 20545 raft_consensus.cc:2454] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7113963474f748cdbf0af9f36ca536d7 in term 1.
I20260619 18:53:28.544998 20806 raft_consensus.cc:2454] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7113963474f748cdbf0af9f36ca536d7 in term 1.
I20260619 18:53:28.545354 20611 leader_election.cc:302] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc9d497340e45b19f010308a7b8872f, 7113963474f748cdbf0af9f36ca536d7; no voters:
I20260619 18:53:28.545727 20875 raft_consensus.cc:2781] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:28.546272 20875 raft_consensus.cc:686] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 1 LEADER]: Becoming Leader. State: Replica: 7113963474f748cdbf0af9f36ca536d7, State: Running, Role: LEADER
I20260619 18:53:28.546883 20875 consensus_queue.cc:237] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc9d497340e45b19f010308a7b8872f" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 34791 } } peers { permanent_uuid: "7113963474f748cdbf0af9f36ca536d7" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 37567 } } peers { permanent_uuid: "4628229a7d9c449b9200000e808be653" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44849 } }
I20260619 18:53:28.552593 20404 catalog_manager.cc:5495] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7113963474f748cdbf0af9f36ca536d7 (127.15.91.2). New cstate: current_term: 1 leader_uuid: "7113963474f748cdbf0af9f36ca536d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc9d497340e45b19f010308a7b8872f" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 34791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7113963474f748cdbf0af9f36ca536d7" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4628229a7d9c449b9200000e808be653" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44849 } health_report { overall_health: UNKNOWN } } }
I20260619 18:53:28.587522 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.585063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.15.91.2:37567)
0619 18:53:28.585297 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:28.585302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:28.585324 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:53:28.585527 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:28.585532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:28.585634 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:28.585726 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:28.586708 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:28.586779 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:28.586786 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:28.586811 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:28.586848 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:28.586861 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:28.586971 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:28.587111 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 18:53:28.587167 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 18:53:28.597148 20806 raft_consensus.cc:1261] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653 [term 1 FOLLOWER]: Refusing update from remote peer 7113963474f748cdbf0af9f36ca536d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:28.597148 20545 raft_consensus.cc:1261] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f [term 1 FOLLOWER]: Refusing update from remote peer 7113963474f748cdbf0af9f36ca536d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:28.598336 20880 consensus_queue.cc:1035] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bc9d497340e45b19f010308a7b8872f" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 34791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:28.598785 20875 consensus_queue.cc:1035] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4628229a7d9c449b9200000e808be653" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:28.612948 20883 mvcc.cc:205] Tried to move back new op lower bound from 7298642774401851392 to 7298642774212640768. Current Snapshot: MvccSnapshot[applied={T|T < 7298642774401851392}]
I20260619 18:53:28.614032 20884 mvcc.cc:205] Tried to move back new op lower bound from 7298642774401851392 to 7298642774212640768. Current Snapshot: MvccSnapshot[applied={T|T < 7298642774401851392}]
I20260619 18:53:28.630095 20404 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:28.630435 20404 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:28.634438 20404 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cc23e38846f49b09cf42732ebf800c0: Sending DeleteTablet for 3 replicas of tablet 931a07f9d7594f28913fe6e2e07f3d1f
I20260619 18:53:28.635869 20525 tablet_service.cc:1514] Processing DeleteTablet for tablet 931a07f9d7594f28913fe6e2e07f3d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:28 UTC) from {username='slave'} at 127.0.0.1:45428
I20260619 18:53:28.636893 20893 tablet_replica.cc:330] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f: stopping tablet replica
I20260619 18:53:28.637056 20786 tablet_service.cc:1514] Processing DeleteTablet for tablet 931a07f9d7594f28913fe6e2e07f3d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:28 UTC) from {username='slave'} at 127.0.0.1:49120
I20260619 18:53:28.637840 20656 tablet_service.cc:1514] Processing DeleteTablet for tablet 931a07f9d7594f28913fe6e2e07f3d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:28 UTC) from {username='slave'} at 127.0.0.1:59994
I20260619 18:53:28.638765 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 20463
I20260619 18:53:28.647969 20893 raft_consensus.cc:2229] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:28.648147 20895 tablet_replica.cc:330] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: stopping tablet replica
I20260619 18:53:28.648991 20893 raft_consensus.cc:2258] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:28.649571 20895 raft_consensus.cc:2229] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:28.651449 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.640160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.15.91.1:34791)
0619 18:53:28.640332 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:28.640338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:28.648464 (+ 8126us) server_negotiation.cc:407] Connection header received
0619 18:53:28.648867 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:28.648873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:28.649060 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:28.649148 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:28.650542 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:28.650548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:28.650554 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:28.650578 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:28.650620 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:28.650637 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:28.650789 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:28.651051 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 18:53:28.651122 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:53:28.652025 20893 ts_tablet_manager.cc:1905] T 931a07f9d7594f28913fe6e2e07f3d1f P 2bc9d497340e45b19f010308a7b8872f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:28.652848 20585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:28.654518 20895 raft_consensus.cc:2258] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:28.655702 20894 tablet_replica.cc:330] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: stopping tablet replica
I20260619 18:53:28.656343 20894 raft_consensus.cc:2229] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:28.657002 20894 raft_consensus.cc:2258] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:28.657756 20895 ts_tablet_manager.cc:1905] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:28.659771 20894 ts_tablet_manager.cc:1905] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:28.667502 20895 ts_tablet_manager.cc:1918] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:28.667804 20895 log.cc:1196] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/931a07f9d7594f28913fe6e2e07f3d1f
I20260619 18:53:28.668466 20895 ts_tablet_manager.cc:1939] T 931a07f9d7594f28913fe6e2e07f3d1f P 7113963474f748cdbf0af9f36ca536d7: Deleting consensus metadata
I20260619 18:53:28.670674 20388 catalog_manager.cc:4841] TS 7113963474f748cdbf0af9f36ca536d7 (127.15.91.2:37567): tablet 931a07f9d7594f28913fe6e2e07f3d1f (table test-table [id=999cbd4b6098460e8a30c323f7598c09]) successfully deleted
I20260619 18:53:28.672052 20894 ts_tablet_manager.cc:1918] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:28.672261 20894 log.cc:1196] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/931a07f9d7594f28913fe6e2e07f3d1f
I20260619 18:53:28.672719 20894 ts_tablet_manager.cc:1939] T 931a07f9d7594f28913fe6e2e07f3d1f P 4628229a7d9c449b9200000e808be653: Deleting consensus metadata
I20260619 18:53:28.674566 20391 catalog_manager.cc:4841] TS 4628229a7d9c449b9200000e808be653 (127.15.91.3:44849): tablet 931a07f9d7594f28913fe6e2e07f3d1f (table test-table [id=999cbd4b6098460e8a30c323f7598c09]) successfully deleted
I20260619 18:53:28.795766 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 20463
I20260619 18:53:28.815742 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 20594
W20260619 18:53:28.815799 20390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.91.1:34791 (error 108)
I20260619 18:53:28.818670 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.817799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:34791 (local address 127.0.0.1:45434)
0619 18:53:28.818368 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:28.818491 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":320,"threads_started":1}
W20260619 18:53:28.819038 20390 catalog_manager.cc:4584] TS 2bc9d497340e45b19f010308a7b8872f (127.15.91.1:34791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 931a07f9d7594f28913fe6e2e07f3d1f: Network error: Client connection negotiation failed: client connection to 127.15.91.1:34791: connect: Connection refused (error 111)
I20260619 18:53:28.819833 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.816714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.15.91.2:37567)
0619 18:53:28.816929 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:28.816936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:28.817399 (+ 463us) server_negotiation.cc:407] Connection header received
0619 18:53:28.817511 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:28.817518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:28.817597 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:28.817824 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:28.818928 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:28.818936 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:28.818942 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:28.818965 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:28.819010 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:28.819035 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:28.819171 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:28.819501 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 18:53:28.819600 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:53:28.820668 20716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:28.872742 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.872191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:34791 (local address 127.0.0.1:45440)
0619 18:53:28.872435 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:28.872562 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 18:53:28.874039 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.873563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:34791 (local address 127.0.0.1:45452)
0619 18:53:28.873735 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:28.873840 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 18:53:28.935040 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.934446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:34791 (local address 127.0.0.1:45462)
0619 18:53:28.934661 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:28.934820 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 18:53:28.935621 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 20594
I20260619 18:53:28.940088 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.939431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:34791 (local address 127.0.0.1:45464)
0619 18:53:28.939686 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:28.939819 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 18:53:28.955776 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 20724
I20260619 18:53:28.961836 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:28.956761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.15.91.3:44849)
0619 18:53:28.956949 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:28.956954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:28.956978 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:53:28.957161 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:28.957166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:28.957263 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:28.957367 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:28.961198 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:28.961203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:28.961207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:28.961232 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:28.961270 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:28.961286 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:28.961427 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:28.961536 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 18:53:28.961593 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:53:28.962581 20846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:29.021082 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:29.020550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:34791 (local address 127.0.0.1:45480)
0619 18:53:29.020815 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:29.020903 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 18:53:29.022275 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:29.021797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:34791 (local address 127.0.0.1:45484)
0619 18:53:29.022019 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:29.022104 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260619 18:53:29.053090 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 20724
I20260619 18:53:29.071396 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 20373
I20260619 18:53:29.072743 20434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:29.180918 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 20373
2026-06-19T18:53:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3076 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 18:53:29.225188 15724 mini_oidc.cc:152] Starting JWKS server
I20260619 18:53:29.228518 15724 webserver.cc:460] Webserver started at https://127.0.0.1:39035/ using document root <none> and password file <none>
I20260619 18:53:29.228960 15724 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:53:29.229801 15724 webserver.cc:460] Webserver started at http://127.0.0.1:38687/ using document root <none> and password file <none>
2026-06-19T18:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:29Z Disabled control of system clock
I20260619 18:53:29.261442 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:40371
--webserver_interface=127.15.91.62
--webserver_port=0
--builtin_ntp_servers=127.15.91.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:40371
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39035/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/testchainca.cert with env {}
W20260619 18:53:29.623241 20908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:29.623575 20908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:29.623760 20908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:29.631880 20908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:29.632005 20908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:29.632090 20908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:29.632201 20908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:29.632265 20908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:53:29.632331 20908 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:53:29.632373 20908 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39035/jwks.json
I20260619 18:53:29.643391 20908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:40371
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:40371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39035/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:29.644891 20908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:29.646308 20908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:29.659957 20914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:29.659993 20916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:29.659780 20913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:29.660986 20908 server_base.cc:1034] running on GCE node
I20260619 18:53:29.661998 20908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:29.663846 20908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:29.665144 20908 hybrid_clock.cc:648] HybridClock initialized: now 1781895209665093 us; error 72 us; skew 500 ppm
I20260619 18:53:29.665725 20908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:53:29.671859 20908 webserver.cc:460] Webserver started at http://127.15.91.62:46109/ using document root <none> and password file <none>
I20260619 18:53:29.672899 20908 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:29.673065 20908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:29.673427 20908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:29.676862 20908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "63bc15e051ad4b399539431e701ca480"
format_stamp: "Formatted at 2026-06-19 18:53:29 on dist-test-slave-dm3f"
server_key: "417049206a81ffb3841873c7b872c292"
server_key_iv: "612fd984ba0b724c5858b1d75f2983d1"
server_key_version: "encryptionkey@0"
I20260619 18:53:29.677840 20908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "63bc15e051ad4b399539431e701ca480"
format_stamp: "Formatted at 2026-06-19 18:53:29 on dist-test-slave-dm3f"
server_key: "417049206a81ffb3841873c7b872c292"
server_key_iv: "612fd984ba0b724c5858b1d75f2983d1"
server_key_version: "encryptionkey@0"
I20260619 18:53:29.684062 20908 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:53:29.688413 20922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:29.690090 20908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:53:29.690347 20908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "63bc15e051ad4b399539431e701ca480"
format_stamp: "Formatted at 2026-06-19 18:53:29 on dist-test-slave-dm3f"
server_key: "417049206a81ffb3841873c7b872c292"
server_key_iv: "612fd984ba0b724c5858b1d75f2983d1"
server_key_version: "encryptionkey@0"
I20260619 18:53:29.690613 20908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:29.710422 20908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:29.711540 20908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:29.711902 20908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:29.734112 20908 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:40371
I20260619 18:53:29.734095 20973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:40371 every 8 connection(s)
I20260619 18:53:29.735813 20908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:29.741237 20974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:29.744444 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 20908
I20260619 18:53:29.744803 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:29.745466 15724 external_mini_cluster.cc:1428] Setting key 6b5a630a40abd599ae3259ed9258e8b8
I20260619 18:53:29.756805 20974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480: Bootstrap starting.
I20260619 18:53:29.761286 20974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:29.762918 20974 log.cc:824] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:29.763922 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:29.747768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.15.91.62:40371)
0619 18:53:29.749026 (+ 1258us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:29.749044 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:29.749108 (+ 64us) server_negotiation.cc:407] Connection header received
0619 18:53:29.751547 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:29.751600 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:29.752433 (+ 833us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:29.753008 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:29.753676 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:29.755790 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:29.757525 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:29.758628 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:29.758956 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:29.760300 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:29.760313 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:29.760335 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:29.761155 (+ 820us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:29.761310 (+ 155us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:29.761348 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:29.761489 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:29.761916 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 18:53:29.762336 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":361,"threads_started":1}
I20260619 18:53:29.766672 20974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480: No bootstrap required, opened a new log
I20260619 18:53:29.768569 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:53:29.775254 20974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bc15e051ad4b399539431e701ca480" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 40371 } }
I20260619 18:53:29.775843 20974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:29.776022 20974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63bc15e051ad4b399539431e701ca480, State: Initialized, Role: FOLLOWER
I20260619 18:53:29.776726 20974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bc15e051ad4b399539431e701ca480" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 40371 } }
I20260619 18:53:29.777015 20974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:29.777170 20974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:29.777370 20974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:29.780072 20974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bc15e051ad4b399539431e701ca480" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 40371 } }
I20260619 18:53:29.780711 20974 leader_election.cc:302] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63bc15e051ad4b399539431e701ca480; no voters:
I20260619 18:53:29.781365 20974 leader_election.cc:288] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:29.781592 20979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:29.782864 20979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [term 1 LEADER]: Becoming Leader. State: Replica: 63bc15e051ad4b399539431e701ca480, State: Running, Role: LEADER
I20260619 18:53:29.783499 20979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bc15e051ad4b399539431e701ca480" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 40371 } }
I20260619 18:53:29.784410 20974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:29.786939 20981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63bc15e051ad4b399539431e701ca480. Latest consensus state: current_term: 1 leader_uuid: "63bc15e051ad4b399539431e701ca480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bc15e051ad4b399539431e701ca480" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 40371 } } }
I20260619 18:53:29.787050 20980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63bc15e051ad4b399539431e701ca480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bc15e051ad4b399539431e701ca480" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 40371 } } }
I20260619 18:53:29.787642 20981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:29.787863 20980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:29.789868 20984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:29.800114 20984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:29.820391 20984 catalog_manager.cc:1347] Generated new cluster ID: e52e5a20d0da428cadf03d53d6a4a388
I20260619 18:53:29.820555 20984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:29.857879 20984 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:29.859517 20984 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:29.869244 20984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63bc15e051ad4b399539431e701ca480: Generated new TSK 0
I20260619 18:53:29.870326 20984 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 18:53:29.922303 21004 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 18:53:29.922518 21004 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:53:29.923905 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:29.900334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.15.91.62:40371)
0619 18:53:29.900595 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:29.900602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:29.901113 (+ 511us) server_negotiation.cc:407] Connection header received
0619 18:53:29.901445 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:29.901452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:29.901563 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:29.901740 (+ 177us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:53:29.902346 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:29.903595 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:29.905326 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:29.906152 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:29.906316 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:29.907188 (+ 872us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:53:29.923133 (+ 15945us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 18:53:29.923572 (+ 439us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46968: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":98}
W20260619 18:53:29.924057 20977 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46968: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 18:53:29.925393 21003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:53:29.900165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:40371 (local address 127.0.0.1:46968)
0619 18:53:29.900977 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:29.900997 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:29.901170 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:29.901777 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:29.901781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:29.901791 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:53:29.902149 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:29.902158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:29.903817 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:29.903822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:29.905090 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:29.905107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:29.906498 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:29.906506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:29.906661 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:29.906820 (+ 159us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:53:29.923514 (+ 16694us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 18:53:29.923815 (+ 301us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.91.62:40371: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":432,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.91.62:40371: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 18:53:29.927170 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 20908
I20260619 18:53:29.928599 20969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:30.026300 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 20908
2026-06-19T18:53:30Z chronyd exiting
I20260619 18:53:30.078189 15724 test_util.cc:175] -----------------------------------------------
I20260619 18:53:30.078356 15724 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781895164554191-15724-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (857 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 18:53:30.082396 15724 mini_oidc.cc:152] Starting JWKS server
I20260619 18:53:30.084493 15724 webserver.cc:460] Webserver started at https://127.0.0.1:40467/ using document root <none> and password file <none>
I20260619 18:53:30.084790 15724 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:53:30.085613 15724 webserver.cc:460] Webserver started at http://127.0.0.1:36893/ using document root <none> and password file <none>
2026-06-19T18:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:30Z Disabled control of system clock
I20260619 18:53:30.115077 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:39161
--webserver_interface=127.15.91.62
--webserver_port=0
--builtin_ntp_servers=127.15.91.20:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:39161
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40467/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/testchainca.cert with env {}
W20260619 18:53:30.473104 21011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:30.473460 21011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:30.473629 21011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:30.481657 21011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:30.481801 21011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:30.481840 21011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:30.481974 21011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:30.482033 21011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:53:30.482164 21011 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:53:30.482221 21011 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40467/jwks.json
I20260619 18:53:30.493521 21011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:39161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40467/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:30.495082 21011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:30.496621 21011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:30.510309 21016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:30.510267 21017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:30.510313 21019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:30.511458 21011 server_base.cc:1034] running on GCE node
I20260619 18:53:30.512403 21011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:30.514213 21011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:30.515476 21011 hybrid_clock.cc:648] HybridClock initialized: now 1781895210515425 us; error 50 us; skew 500 ppm
I20260619 18:53:30.515959 21011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:53:30.518968 21011 webserver.cc:460] Webserver started at http://127.15.91.62:41675/ using document root <none> and password file <none>
I20260619 18:53:30.519845 21011 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:30.520009 21011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:30.520393 21011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:30.523804 21011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "99881f127c654660b5bd1616e97dbb3a"
format_stamp: "Formatted at 2026-06-19 18:53:30 on dist-test-slave-dm3f"
server_key: "4cb8453a5e3b4fff19f23e2c1025a077"
server_key_iv: "7bf3c07d20ec451904d47759fab5fcea"
server_key_version: "encryptionkey@0"
I20260619 18:53:30.524808 21011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "99881f127c654660b5bd1616e97dbb3a"
format_stamp: "Formatted at 2026-06-19 18:53:30 on dist-test-slave-dm3f"
server_key: "4cb8453a5e3b4fff19f23e2c1025a077"
server_key_iv: "7bf3c07d20ec451904d47759fab5fcea"
server_key_version: "encryptionkey@0"
I20260619 18:53:30.531443 21011 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 18:53:30.535952 21025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:30.537900 21011 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:53:30.538189 21011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "99881f127c654660b5bd1616e97dbb3a"
format_stamp: "Formatted at 2026-06-19 18:53:30 on dist-test-slave-dm3f"
server_key: "4cb8453a5e3b4fff19f23e2c1025a077"
server_key_iv: "7bf3c07d20ec451904d47759fab5fcea"
server_key_version: "encryptionkey@0"
I20260619 18:53:30.538498 21011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:30.552115 21011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:30.553123 21011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:30.553483 21011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:30.574566 21011 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:39161
I20260619 18:53:30.574585 21076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:39161 every 8 connection(s)
I20260619 18:53:30.576184 21011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:30.578003 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 21011
I20260619 18:53:30.578219 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:30.579092 15724 external_mini_cluster.cc:1428] Setting key 66926f10741165d533d814063a0f8a5d
I20260619 18:53:30.583233 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:30.594578 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:30.581118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.15.91.62:39161)
0619 18:53:30.582222 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:30.582236 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:30.582287 (+ 51us) server_negotiation.cc:407] Connection header received
0619 18:53:30.584164 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:30.584192 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:30.584757 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:30.585223 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:30.585935 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:30.587595 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:30.589087 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:30.589909 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:30.590149 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:30.591714 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:30.591723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:30.591742 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:30.592398 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:30.592500 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:30.592527 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:30.592707 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:30.593021 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 18:53:30.593417 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":356,"threads_started":1}
I20260619 18:53:30.596472 21077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a: Bootstrap starting.
I20260619 18:53:30.599388 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:53:30.600533 21077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:30.601826 21077 log.cc:824] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:30.605366 21077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a: No bootstrap required, opened a new log
I20260619 18:53:30.613368 21077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99881f127c654660b5bd1616e97dbb3a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39161 } }
I20260619 18:53:30.613936 21077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:30.614085 21077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99881f127c654660b5bd1616e97dbb3a, State: Initialized, Role: FOLLOWER
I20260619 18:53:30.614907 21077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99881f127c654660b5bd1616e97dbb3a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39161 } }
I20260619 18:53:30.615243 21077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:30.615429 21077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:30.615638 21077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:30.618041 21077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99881f127c654660b5bd1616e97dbb3a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39161 } }
I20260619 18:53:30.618645 21077 leader_election.cc:302] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99881f127c654660b5bd1616e97dbb3a; no voters:
I20260619 18:53:30.619383 21077 leader_election.cc:288] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:30.619591 21082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:30.621026 21082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [term 1 LEADER]: Becoming Leader. State: Replica: 99881f127c654660b5bd1616e97dbb3a, State: Running, Role: LEADER
I20260619 18:53:30.621755 21082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99881f127c654660b5bd1616e97dbb3a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39161 } }
I20260619 18:53:30.622057 21077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:30.625487 21084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99881f127c654660b5bd1616e97dbb3a. Latest consensus state: current_term: 1 leader_uuid: "99881f127c654660b5bd1616e97dbb3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99881f127c654660b5bd1616e97dbb3a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39161 } } }
I20260619 18:53:30.625577 21083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99881f127c654660b5bd1616e97dbb3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99881f127c654660b5bd1616e97dbb3a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39161 } } }
I20260619 18:53:30.625924 21084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:30.626000 21083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:30.628654 21090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:30.635309 21090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:30.646363 21090 catalog_manager.cc:1347] Generated new cluster ID: 0db97428f2bb440b9d9b33232b7c0aaa
I20260619 18:53:30.646518 21090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:30.690416 21090 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:30.692030 21090 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:30.700038 21090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99881f127c654660b5bd1616e97dbb3a: Generated new TSK 0
I20260619 18:53:30.701089 21090 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 18:53:30.742524 21107 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 18:53:30.742674 21107 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:53:30.744493 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:30.721051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.15.91.62:39161)
0619 18:53:30.721274 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:30.721281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:30.721643 (+ 362us) server_negotiation.cc:407] Connection header received
0619 18:53:30.721896 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:30.721903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:30.722018 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:30.722234 (+ 216us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:53:30.722692 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:30.723888 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:30.725529 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:30.726596 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:30.726936 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:30.727728 (+ 792us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:53:30.743630 (+ 15902us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 18:53:30.744138 (+ 508us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38958: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":89}
W20260619 18:53:30.744663 21080 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38958: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 18:53:30.744797 21106 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:53:30.720819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39161 (local address 127.0.0.1:38958)
0619 18:53:30.721515 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:30.721534 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:30.721681 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:30.722247 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:30.722250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:30.722261 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:53:30.722521 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:30.722530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:30.724084 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:30.724089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:30.725245 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:30.725259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:30.727013 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:30.727019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:30.727210 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:30.727257 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:53:30.744137 (+ 16880us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 18:53:30.744405 (+ 268us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.91.62:39161: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":379,"threads_started":1}
I20260619 18:53:30.745956 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 21011
I20260619 18:53:30.747437 21072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:30.849246 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 21011
2026-06-19T18:53:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (823 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 18:53:30.906453 15724 mini_oidc.cc:152] Starting JWKS server
I20260619 18:53:30.908527 15724 webserver.cc:460] Webserver started at https://127.0.0.1:40575/ using document root <none> and password file <none>
I20260619 18:53:30.908776 15724 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:53:30.909477 15724 webserver.cc:460] Webserver started at http://127.0.0.1:43161/ using document root <none> and password file <none>
2026-06-19T18:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:30Z Disabled control of system clock
I20260619 18:53:30.939464 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:34731
--webserver_interface=127.15.91.62
--webserver_port=0
--builtin_ntp_servers=127.15.91.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:34731
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40575/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 18:53:31.296176 21114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:31.296572 21114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:31.296742 21114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:31.304512 21114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:31.304654 21114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:31.304693 21114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:31.304818 21114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:31.304908 21114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:53:31.305038 21114 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:53:31.305094 21114 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40575/jwks.json
I20260619 18:53:31.315269 21114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:34731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:34731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40575/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:31.316816 21114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:31.318910 21114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:31.334189 21119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:31.334142 21120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:31.334231 21122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:31.335142 21114 server_base.cc:1034] running on GCE node
I20260619 18:53:31.336122 21114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:31.337936 21114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:31.339237 21114 hybrid_clock.cc:648] HybridClock initialized: now 1781895211339198 us; error 75 us; skew 500 ppm
I20260619 18:53:31.339684 21114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:53:31.342600 21114 webserver.cc:460] Webserver started at http://127.15.91.62:36999/ using document root <none> and password file <none>
I20260619 18:53:31.343523 21114 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:31.343695 21114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:31.344076 21114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:31.347510 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "2cd81a2af21146ff93e80fe5f2d9bd4b"
format_stamp: "Formatted at 2026-06-19 18:53:31 on dist-test-slave-dm3f"
server_key: "04ea63d11a6e7b507f3f1d5cfc123135"
server_key_iv: "c77d2a8cde5f1c6c21668461c52b146f"
server_key_version: "encryptionkey@0"
I20260619 18:53:31.348519 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "2cd81a2af21146ff93e80fe5f2d9bd4b"
format_stamp: "Formatted at 2026-06-19 18:53:31 on dist-test-slave-dm3f"
server_key: "04ea63d11a6e7b507f3f1d5cfc123135"
server_key_iv: "c77d2a8cde5f1c6c21668461c52b146f"
server_key_version: "encryptionkey@0"
I20260619 18:53:31.354928 21114 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:53:31.359241 21128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:31.360937 21114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:53:31.361217 21114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "2cd81a2af21146ff93e80fe5f2d9bd4b"
format_stamp: "Formatted at 2026-06-19 18:53:31 on dist-test-slave-dm3f"
server_key: "04ea63d11a6e7b507f3f1d5cfc123135"
server_key_iv: "c77d2a8cde5f1c6c21668461c52b146f"
server_key_version: "encryptionkey@0"
I20260619 18:53:31.361505 21114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:31.396412 21114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:31.397385 21114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:31.397740 21114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:31.419435 21114 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:34731
I20260619 18:53:31.419453 21179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:34731 every 8 connection(s)
I20260619 18:53:31.421048 21114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:31.422981 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 21114
I20260619 18:53:31.423218 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:31.424052 15724 external_mini_cluster.cc:1428] Setting key 2ec049fb3044517a55153776d6381b1f
I20260619 18:53:31.427287 21180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:31.440099 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b: Bootstrap starting.
I20260619 18:53:31.440232 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:31.426756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.15.91.62:34731)
0619 18:53:31.428015 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:31.428036 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:31.428088 (+ 52us) server_negotiation.cc:407] Connection header received
0619 18:53:31.429907 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:31.429935 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:31.430556 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:31.431161 (+ 605us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:31.431868 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:31.433409 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:31.435057 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:31.435835 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:31.436092 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:31.437408 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:31.437421 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:31.437439 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:31.438110 (+ 671us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:31.438201 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:31.438228 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:31.438375 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:31.438635 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 18:53:31.439055 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":413,"threads_started":1}
I20260619 18:53:31.444413 21180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:31.444761 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:53:31.445690 21180 log.cc:824] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:31.449331 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b: No bootstrap required, opened a new log
I20260619 18:53:31.458438 21180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd81a2af21146ff93e80fe5f2d9bd4b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34731 } }
I20260619 18:53:31.458967 21180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:31.459097 21180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd81a2af21146ff93e80fe5f2d9bd4b, State: Initialized, Role: FOLLOWER
I20260619 18:53:31.459810 21180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd81a2af21146ff93e80fe5f2d9bd4b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34731 } }
I20260619 18:53:31.460098 21180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:31.460246 21180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:31.460458 21180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:31.463321 21180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd81a2af21146ff93e80fe5f2d9bd4b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34731 } }
I20260619 18:53:31.463936 21180 leader_election.cc:302] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cd81a2af21146ff93e80fe5f2d9bd4b; no voters:
I20260619 18:53:31.464596 21180 leader_election.cc:288] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:31.464809 21185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:31.465696 21185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [term 1 LEADER]: Becoming Leader. State: Replica: 2cd81a2af21146ff93e80fe5f2d9bd4b, State: Running, Role: LEADER
I20260619 18:53:31.466349 21185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd81a2af21146ff93e80fe5f2d9bd4b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34731 } }
I20260619 18:53:31.467211 21180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:31.469684 21187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd81a2af21146ff93e80fe5f2d9bd4b. Latest consensus state: current_term: 1 leader_uuid: "2cd81a2af21146ff93e80fe5f2d9bd4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd81a2af21146ff93e80fe5f2d9bd4b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34731 } } }
I20260619 18:53:31.470247 21187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:31.470548 21186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cd81a2af21146ff93e80fe5f2d9bd4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd81a2af21146ff93e80fe5f2d9bd4b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34731 } } }
I20260619 18:53:31.470973 21186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:31.471812 21194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:31.478014 21194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:31.488713 21194 catalog_manager.cc:1347] Generated new cluster ID: b074832e19694422b9299229814332ad
I20260619 18:53:31.488854 21194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:31.507217 21194 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:31.509333 21194 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:31.524842 21194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cd81a2af21146ff93e80fe5f2d9bd4b: Generated new TSK 0
I20260619 18:53:31.526232 21194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:53:31.551540 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:31.541730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.15.91.62:34731)
0619 18:53:31.541988 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:31.541995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:31.542416 (+ 421us) server_negotiation.cc:407] Connection header received
0619 18:53:31.542687 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:31.542691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:31.542830 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:31.542981 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:53:31.543582 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:31.544359 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:31.546541 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:31.547171 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:31.547292 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:31.548060 (+ 768us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:53:31.550786 (+ 2726us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 18:53:31.551180 (+ 394us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 18:53:31.551689 21183 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 18:53:31.553727 21209 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:53:31.541536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34731 (local address 127.0.0.1:41262)
0619 18:53:31.542269 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:31.542292 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:31.542489 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:31.543029 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:31.543035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:31.543049 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:53:31.543395 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:31.543407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:31.544532 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:31.544538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:31.546288 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:31.546308 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:31.547423 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:31.547431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:31.547647 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:31.547706 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:53:31.551072 (+ 3366us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 18:53:31.553248 (+ 2176us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.91.62:34731: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":1722,"thread_start_us":381,"threads_started":1}
I20260619 18:53:31.555127 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 21114
I20260619 18:53:31.557113 21175 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 18:53:31.564592 21210 webserver.cc:533] Webserver: sslize failed:
E20260619 18:53:31.564745 21210 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:53:31.650940 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 21114
2026-06-19T18:53:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (801 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T18:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:31Z Disabled control of system clock
I20260619 18:53:31.737133 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:43171
--webserver_interface=127.15.91.62
--webserver_port=0
--builtin_ntp_servers=127.15.91.20:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:43171
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/testchainca.cert with env {}
W20260619 18:53:32.096596 21215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:32.096937 21215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:32.097036 21215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:32.104846 21215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:32.104995 21215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:32.105036 21215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:32.105085 21215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:32.105131 21215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:53:32.105188 21215 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:53:32.105228 21215 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 18:53:32.116043 21215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:43171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:43171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:32.117434 21215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:32.119027 21215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:32.132278 21221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:32.132402 21220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:32.132436 21223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:32.133737 21215 server_base.cc:1034] running on GCE node
I20260619 18:53:32.134812 21215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:32.136679 21215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:32.137970 21215 hybrid_clock.cc:648] HybridClock initialized: now 1781895212137908 us; error 74 us; skew 500 ppm
I20260619 18:53:32.138401 21215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:53:32.141240 21215 webserver.cc:460] Webserver started at http://127.15.91.62:45375/ using document root <none> and password file <none>
I20260619 18:53:32.142107 21215 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:32.142285 21215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:32.142642 21215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:32.145982 21215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "b5361b894e01430b912f3e05f13035b7"
format_stamp: "Formatted at 2026-06-19 18:53:32 on dist-test-slave-dm3f"
server_key: "ca5505027cc84d6de11e945d32e7c206"
server_key_iv: "355db36973f327989b337b7c7dbb3cac"
server_key_version: "encryptionkey@0"
I20260619 18:53:32.147107 21215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "b5361b894e01430b912f3e05f13035b7"
format_stamp: "Formatted at 2026-06-19 18:53:32 on dist-test-slave-dm3f"
server_key: "ca5505027cc84d6de11e945d32e7c206"
server_key_iv: "355db36973f327989b337b7c7dbb3cac"
server_key_version: "encryptionkey@0"
I20260619 18:53:32.153365 21215 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:53:32.157768 21229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:32.159619 21215 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:53:32.159896 21215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "b5361b894e01430b912f3e05f13035b7"
format_stamp: "Formatted at 2026-06-19 18:53:32 on dist-test-slave-dm3f"
server_key: "ca5505027cc84d6de11e945d32e7c206"
server_key_iv: "355db36973f327989b337b7c7dbb3cac"
server_key_version: "encryptionkey@0"
I20260619 18:53:32.160182 21215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:32.178263 21215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:32.179328 21215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:32.179675 21215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:32.200506 21215 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:43171
I20260619 18:53:32.200520 21280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:43171 every 8 connection(s)
I20260619 18:53:32.202126 21215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:32.206498 21281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:32.210292 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 21215
I20260619 18:53:32.210489 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:32.211236 15724 external_mini_cluster.cc:1428] Setting key e07f2f2856e26747cb34be7718cde82c
I20260619 18:53:32.221093 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7: Bootstrap starting.
I20260619 18:53:32.225492 21281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:32.226836 21281 log.cc:824] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:32.227124 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:32.213463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.15.91.62:43171)
0619 18:53:32.214576 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:32.214590 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:32.214639 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:53:32.216419 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:32.216449 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:32.217066 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:32.217479 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:32.218201 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:32.219732 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:32.221717 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:32.222509 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:32.222810 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:32.224195 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:32.224204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:32.224221 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:53:32.224834 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:32.224941 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:53:32.224971 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:32.225076 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:32.225364 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 18:53:32.225854 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":384,"threads_started":1}
I20260619 18:53:32.230268 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7: No bootstrap required, opened a new log
I20260619 18:53:32.231729 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:53:32.238899 21281 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5361b894e01430b912f3e05f13035b7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43171 } }
I20260619 18:53:32.239472 21281 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:32.239641 21281 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5361b894e01430b912f3e05f13035b7, State: Initialized, Role: FOLLOWER
I20260619 18:53:32.240412 21281 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5361b894e01430b912f3e05f13035b7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43171 } }
I20260619 18:53:32.240715 21281 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:32.240870 21281 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:32.241065 21281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:32.243603 21281 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5361b894e01430b912f3e05f13035b7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43171 } }
I20260619 18:53:32.244284 21281 leader_election.cc:302] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5361b894e01430b912f3e05f13035b7; no voters:
I20260619 18:53:32.245030 21281 leader_election.cc:288] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:32.245198 21286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:32.246479 21286 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [term 1 LEADER]: Becoming Leader. State: Replica: b5361b894e01430b912f3e05f13035b7, State: Running, Role: LEADER
I20260619 18:53:32.247140 21286 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5361b894e01430b912f3e05f13035b7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43171 } }
I20260619 18:53:32.247560 21281 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:32.250988 21288 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5361b894e01430b912f3e05f13035b7. Latest consensus state: current_term: 1 leader_uuid: "b5361b894e01430b912f3e05f13035b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5361b894e01430b912f3e05f13035b7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43171 } } }
I20260619 18:53:32.251411 21288 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:32.251657 21287 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5361b894e01430b912f3e05f13035b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5361b894e01430b912f3e05f13035b7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43171 } } }
I20260619 18:53:32.252059 21287 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:32.254649 21294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:32.261147 21294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:32.272421 21294 catalog_manager.cc:1347] Generated new cluster ID: 304b732e582347af90e4a22fbac80134
I20260619 18:53:32.272583 21294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:32.291694 21294 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:32.293144 21294 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:32.303938 21294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5361b894e01430b912f3e05f13035b7: Generated new TSK 0
I20260619 18:53:32.304982 21294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:53:32.405725 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:32.325198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.15.91.62:43171)
0619 18:53:32.325494 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:32.325505 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:32.326156 (+ 651us) server_negotiation.cc:407] Connection header received
0619 18:53:32.326466 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:32.326471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:32.326560 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:32.326763 (+ 203us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:53:32.327291 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:32.328076 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:32.329591 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:32.330076 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:32.330243 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:32.330851 (+ 608us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:53:32.404858 (+ 74007us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 18:53:32.405348 (+ 490us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46526: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":101}
W20260619 18:53:32.405908 21310 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:53:32.324971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43171 (local address 127.0.0.1:46526)
0619 18:53:32.326018 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:32.326036 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:32.326235 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:32.326785 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:32.326789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:32.326801 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:53:32.327116 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:32.327125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:32.328256 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:32.328262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:32.329409 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:32.329421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:32.330273 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:32.330280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:32.330449 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:32.330489 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:53:32.405189 (+ 74700us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 18:53:32.405518 (+ 329us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.91.62:43171: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":725,"threads_started":1}
W20260619 18:53:32.405908 21284 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46526: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260619 18:53:32.407189 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 21215
I20260619 18:53:32.408717 21276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:32.513080 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 21215
2026-06-19T18:53:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (841 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (444 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (455 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:53:33 dist-test-slave-dm3f krb5kdc[21325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:53:33 dist-test-slave-dm3f krb5kdc[21325](info): set up 2 sockets
Jun 19 18:53:33 dist-test-slave-dm3f krb5kdc[21325](info): commencing operation
krb5kdc: starting...
W20260619 18:53:34.499883 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:53:34 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:53:34.724525 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:45449
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:45449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:35.081977 21338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:35.082316 21338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:35.082494 21338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:35.090196 21338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:35.090338 21338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:35.090433 21338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:35.090526 21338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:35.090608 21338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:35.101977 21338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:45449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:45449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:35.103523 21338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:35.104990 21338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:35.118697 21344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:35.118745 21343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:35.118649 21346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:35.119623 21338 server_base.cc:1034] running on GCE node
I20260619 18:53:35.120543 21338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:35.122191 21338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:35.123467 21338 hybrid_clock.cc:648] HybridClock initialized: now 1781895215123422 us; error 76 us; skew 500 ppm
I20260619 18:53:35.123910 21338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:35 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895215, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:35.131537 21338 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:53:35.133471 21338 webserver.cc:460] Webserver started at http://127.15.91.62:34299/ using document root <none> and password file <none>
I20260619 18:53:35.134316 21338 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:35.134497 21338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:35.134936 21338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:35.138291 21338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "bac3841f86f54a77bea58a4c960fef88"
format_stamp: "Formatted at 2026-06-19 18:53:35 on dist-test-slave-dm3f"
server_key: "011907a4c99ce9c8df1c9785fdac6501"
server_key_iv: "9c17630bbe8a1f878ddc3e480dcf81f5"
server_key_version: "encryptionkey@0"
I20260619 18:53:35.139284 21338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "bac3841f86f54a77bea58a4c960fef88"
format_stamp: "Formatted at 2026-06-19 18:53:35 on dist-test-slave-dm3f"
server_key: "011907a4c99ce9c8df1c9785fdac6501"
server_key_iv: "9c17630bbe8a1f878ddc3e480dcf81f5"
server_key_version: "encryptionkey@0"
I20260619 18:53:35.145494 21338 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:53:35.149775 21353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:35.151558 21338 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:53:35.151821 21338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "bac3841f86f54a77bea58a4c960fef88"
format_stamp: "Formatted at 2026-06-19 18:53:35 on dist-test-slave-dm3f"
server_key: "011907a4c99ce9c8df1c9785fdac6501"
server_key_iv: "9c17630bbe8a1f878ddc3e480dcf81f5"
server_key_version: "encryptionkey@0"
I20260619 18:53:35.152067 21338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:35.167474 21338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:35.171941 21338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:35.172283 21338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:35.193364 21338 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:45449
I20260619 18:53:35.193401 21404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:45449 every 8 connection(s)
I20260619 18:53:35.195014 21338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:35.197933 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 21338
I20260619 18:53:35.198158 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:35.199033 15724 external_mini_cluster.cc:1428] Setting key 2b332d8ee3b6c3e2f536bdafd7864f2b
I20260619 18:53:35.200305 21405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:53:35 dist-test-slave-dm3f krb5kdc[21325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:35.217113 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88: Bootstrap starting.
I20260619 18:53:35.221923 21405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:35.223414 21405 log.cc:824] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:35.227325 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:35.201541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.15.91.62:45449)
0619 18:53:35.202473 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:35.202487 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:35.202534 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:53:35.204327 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:35.204360 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:35.204906 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:35.205314 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:35.207382 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:35.208893 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:35.210846 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:35.211608 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:35.211777 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:35.217004 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:35.217018 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:35.217042 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:35.217109 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:35.221460 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:35.222545 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:35.222556 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:35.222565 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:35.222859 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:35.223341 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:35.223350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:35.223356 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:35.224251 (+ 895us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:35.224529 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:35.225267 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 18:53:35.225817 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":438,"threads_started":1}
I20260619 18:53:35.227511 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88: No bootstrap required, opened a new log
I20260619 18:53:35.236510 21405 raft_consensus.cc:348] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac3841f86f54a77bea58a4c960fef88" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 45449 } }
I20260619 18:53:35.236949 21405 raft_consensus.cc:374] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:35.237033 21405 raft_consensus.cc:729] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac3841f86f54a77bea58a4c960fef88, State: Initialized, Role: FOLLOWER
I20260619 18:53:35.237604 21405 consensus_queue.cc:260] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac3841f86f54a77bea58a4c960fef88" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 45449 } }
I20260619 18:53:35.237829 21405 raft_consensus.cc:388] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:35.237937 21405 raft_consensus.cc:482] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:35.238085 21405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:35.240561 21405 raft_consensus.cc:504] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac3841f86f54a77bea58a4c960fef88" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 45449 } }
I20260619 18:53:35.241127 21405 leader_election.cc:302] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bac3841f86f54a77bea58a4c960fef88; no voters:
I20260619 18:53:35.241864 21405 leader_election.cc:288] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:35.242040 21410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:35.243398 21410 raft_consensus.cc:686] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [term 1 LEADER]: Becoming Leader. State: Replica: bac3841f86f54a77bea58a4c960fef88, State: Running, Role: LEADER
I20260619 18:53:35.243990 21410 consensus_queue.cc:237] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac3841f86f54a77bea58a4c960fef88" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 45449 } }
I20260619 18:53:35.244513 21405 sys_catalog.cc:564] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:35.247040 21412 sys_catalog.cc:455] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bac3841f86f54a77bea58a4c960fef88. Latest consensus state: current_term: 1 leader_uuid: "bac3841f86f54a77bea58a4c960fef88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac3841f86f54a77bea58a4c960fef88" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 45449 } } }
I20260619 18:53:35.247529 21412 sys_catalog.cc:458] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:35.248848 21411 sys_catalog.cc:455] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bac3841f86f54a77bea58a4c960fef88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac3841f86f54a77bea58a4c960fef88" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 45449 } } }
I20260619 18:53:35.249292 21411 sys_catalog.cc:458] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:35.253142 21418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:35.259635 21418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:35.272796 21418 catalog_manager.cc:1347] Generated new cluster ID: 762441f61d0642c5bcfb2e1e3736229b
I20260619 18:53:35.272960 21418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:35.301980 21418 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:35.303678 21418 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:35.320212 21418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bac3841f86f54a77bea58a4c960fef88: Generated new TSK 0
I20260619 18:53:35.321272 21418 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:53:35.380775 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:45449
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:35.765802 21431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:35.766197 21431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:35.766373 21431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:35.774573 21431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:35.774789 21431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:35.774996 21431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:53:35.785733 21431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:45449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:35.787256 21431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:35.788734 21431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:35.804121 21436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:35.804090 21437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:35.804342 21439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:35.805107 21431 server_base.cc:1034] running on GCE node
I20260619 18:53:35.805866 21431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:35.807137 21431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:35.808399 21431 hybrid_clock.cc:648] HybridClock initialized: now 1781895215808321 us; error 73 us; skew 500 ppm
I20260619 18:53:35.808750 21431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:35 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895215, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:35.812515 21431 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:53:35.814358 21431 webserver.cc:460] Webserver started at http://127.15.91.1:34457/ using document root <none> and password file <none>
I20260619 18:53:35.815255 21431 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:35.815451 21431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:35.815804 21431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:35.819388 21431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "ddafa576215a4044a1b304cf95ed4dc5"
format_stamp: "Formatted at 2026-06-19 18:53:35 on dist-test-slave-dm3f"
server_key: "85b1becd874efc70b46e429e9047dcb2"
server_key_iv: "8ece5e6d84d8dbdafb6e51d87e23c8dc"
server_key_version: "encryptionkey@0"
I20260619 18:53:35.820441 21431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "ddafa576215a4044a1b304cf95ed4dc5"
format_stamp: "Formatted at 2026-06-19 18:53:35 on dist-test-slave-dm3f"
server_key: "85b1becd874efc70b46e429e9047dcb2"
server_key_iv: "8ece5e6d84d8dbdafb6e51d87e23c8dc"
server_key_version: "encryptionkey@0"
I20260619 18:53:35.826901 21431 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:53:35.831647 21446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:35.833958 21431 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260619 18:53:35.834266 21431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "ddafa576215a4044a1b304cf95ed4dc5"
format_stamp: "Formatted at 2026-06-19 18:53:35 on dist-test-slave-dm3f"
server_key: "85b1becd874efc70b46e429e9047dcb2"
server_key_iv: "8ece5e6d84d8dbdafb6e51d87e23c8dc"
server_key_version: "encryptionkey@0"
I20260619 18:53:35.834558 21431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:35.855489 21431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:35.860491 21431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:35.860827 21431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:35.861932 21431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:35.863644 21431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:35.863775 21431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:35.863955 21431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:35.864033 21431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:35.901604 21431 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:33531
I20260619 18:53:35.901655 21558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:33531 every 8 connection(s)
I20260619 18:53:35.903165 21431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:53:35.909453 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 21431
I20260619 18:53:35.909672 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:53:35.910352 15724 external_mini_cluster.cc:1428] Setting key af9b94e7ad64d65a9e4468b4ba6df698
Jun 19 18:53:35 dist-test-slave-dm3f krb5kdc[21325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895215, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:35.928202 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:35.906123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:53183 (local address 127.15.91.62:45449)
0619 18:53:35.906629 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:35.906634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:35.907697 (+ 1063us) server_negotiation.cc:407] Connection header received
0619 18:53:35.909797 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:35.909803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:35.909933 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:35.910069 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:35.912201 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:35.912966 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:35.914781 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:35.915218 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:35.915381 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:35.919752 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:35.919762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:35.919771 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:35.919855 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:35.923838 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:35.924900 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:35.924910 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:35.924917 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:35.925103 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:35.925696 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:35.925705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:35.925711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:35.926174 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:35.926354 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:35.927371 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0619 18:53:35.927678 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":275,"threads_started":1}
I20260619 18:53:35.928999 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:35.906592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:45449 (local address 127.15.91.1:53183)
0619 18:53:35.907498 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:35.907549 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:35.909403 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:35.910361 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:35.910372 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:35.911171 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:35.911994 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:35.912016 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:35.913175 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:35.913180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:35.914545 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:35.914562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:35.915527 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:35.915535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:35.915811 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:35.916636 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:35.916662 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:35.919364 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:35.924107 (+ 4743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:35.924117 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:35.924135 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:35.924686 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:35.925271 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:35.925276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:35.925278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:35.925532 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:35.926364 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:35.926372 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:35.926892 (+ 520us) client_negotiation.cc:769] Sending connection context
0619 18:53:35.927177 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 18:53:35.927654 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":329,"threads_started":1}
I20260619 18:53:35.931088 21559 heartbeater.cc:347] Connected to a master server at 127.15.91.62:45449
I20260619 18:53:35.931512 21559 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:53:35.932639 21559 heartbeater.cc:511] Master 127.15.91.62:45449 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:53:35.935642 21370 ts_manager.cc:194] Registered new tserver with Master: ddafa576215a4044a1b304cf95ed4dc5 (127.15.91.1:33531)
I20260619 18:53:35.938151 21370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:53183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:53:35.961862 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:45449
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:36.314831 21565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:36.315209 21565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:36.315410 21565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:36.323926 21565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:36.324116 21565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:36.324299 21565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:53:36.335180 21565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:45449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:36.336690 21565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:36.338174 21565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:36.353446 21571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:36.353648 21570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:36.354027 21573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:36.354493 21565 server_base.cc:1034] running on GCE node
I20260619 18:53:36.355302 21565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:36.356560 21565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:36.357801 21565 hybrid_clock.cc:648] HybridClock initialized: now 1781895216357714 us; error 129 us; skew 500 ppm
I20260619 18:53:36.358197 21565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:36 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:36.361769 21565 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:53:36.363627 21565 webserver.cc:460] Webserver started at http://127.15.91.2:35683/ using document root <none> and password file <none>
I20260619 18:53:36.364459 21565 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:36.364634 21565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:36.364998 21565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:36.368403 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "10c9c02b15f545d0a481bf8ba52d0106"
format_stamp: "Formatted at 2026-06-19 18:53:36 on dist-test-slave-dm3f"
server_key: "15b74f838a0ed714cd6d7ef73033328e"
server_key_iv: "6e33ade5a027a76cc30e965ea2be95b5"
server_key_version: "encryptionkey@0"
I20260619 18:53:36.369345 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "10c9c02b15f545d0a481bf8ba52d0106"
format_stamp: "Formatted at 2026-06-19 18:53:36 on dist-test-slave-dm3f"
server_key: "15b74f838a0ed714cd6d7ef73033328e"
server_key_iv: "6e33ade5a027a76cc30e965ea2be95b5"
server_key_version: "encryptionkey@0"
I20260619 18:53:36.375588 21565 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 18:53:36.379882 21580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:36.381754 21565 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260619 18:53:36.382000 21565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "10c9c02b15f545d0a481bf8ba52d0106"
format_stamp: "Formatted at 2026-06-19 18:53:36 on dist-test-slave-dm3f"
server_key: "15b74f838a0ed714cd6d7ef73033328e"
server_key_iv: "6e33ade5a027a76cc30e965ea2be95b5"
server_key_version: "encryptionkey@0"
I20260619 18:53:36.382262 21565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:36.404203 21565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:36.409159 21565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:36.409525 21565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:36.410864 21565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:36.412647 21565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:36.412784 21565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:36.412966 21565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:36.413136 21565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:36.451714 21565 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:34941
I20260619 18:53:36.451749 21692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:34941 every 8 connection(s)
I20260619 18:53:36.453239 21565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:53:36.455497 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 21565
I20260619 18:53:36.455693 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:53:36.456281 15724 external_mini_cluster.cc:1428] Setting key 3f9d65a9a024fd3ee74754dd1a1918a4
Jun 19 18:53:36 dist-test-slave-dm3f krb5kdc[21325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
I20260619 18:53:36.477257 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:36.456528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:50267 (local address 127.15.91.62:45449)
0619 18:53:36.457029 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:36.457034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:36.458634 (+ 1600us) server_negotiation.cc:407] Connection header received
0619 18:53:36.460724 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:36.460730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:36.460836 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:36.460964 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:36.462787 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:36.463555 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:36.465355 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:36.465862 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:36.466027 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:36.470327 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:36.470333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:36.470336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:36.470376 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:36.473399 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:36.474373 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:36.474380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:36.474384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:36.474578 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:36.475152 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:36.475158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:36.475163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:36.475474 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:36.475661 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:36.476575 (+ 914us) server_negotiation.cc:299] Negotiation successful
0619 18:53:36.476842 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":268,"threads_started":1}
I20260619 18:53:36.478374 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:36.457367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:45449 (local address 127.15.91.2:50267)
0619 18:53:36.458432 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:36.458500 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:36.460367 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:36.461267 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:36.461276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:36.461879 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:36.462516 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:36.462536 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:36.463753 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:36.463759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:36.465141 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:36.465158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:36.466169 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:36.466178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:36.466443 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:36.467282 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:36.467311 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:36.469971 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:36.473637 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:36.473647 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:36.473665 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:36.474202 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:36.474746 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:36.474754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:36.474761 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:36.475017 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:36.475636 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:36.475646 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:36.476113 (+ 467us) client_negotiation.cc:769] Sending connection context
0619 18:53:36.476454 (+ 341us) client_negotiation.cc:240] Negotiation successful
0619 18:53:36.477077 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":391,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
I20260619 18:53:36.480211 21693 heartbeater.cc:347] Connected to a master server at 127.15.91.62:45449
I20260619 18:53:36.480629 21693 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:36.481688 21693 heartbeater.cc:511] Master 127.15.91.62:45449 requested a full tablet report, sending...
I20260619 18:53:36.483760 21370 ts_manager.cc:194] Registered new tserver with Master: 10c9c02b15f545d0a481bf8ba52d0106 (127.15.91.2:34941)
I20260619 18:53:36.484683 21370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:50267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:53:36.506240 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:45449
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:36.862293 21699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:36.862685 21699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:36.862890 21699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:36.870747 21699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:36.870954 21699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:36.871147 21699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:53:36.881624 21699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:45449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:36.883258 21699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:36.884737 21699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:36.899977 21704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:36.899923 21707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:36.899887 21705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:36.902999 21699 server_base.cc:1034] running on GCE node
I20260619 18:53:36.903843 21699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:36.905085 21699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:36.906383 21699 hybrid_clock.cc:648] HybridClock initialized: now 1781895216906328 us; error 52 us; skew 500 ppm
I20260619 18:53:36.906857 21699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:36 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:36.910414 21699 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:53:36.912384 21699 webserver.cc:460] Webserver started at http://127.15.91.3:43323/ using document root <none> and password file <none>
I20260619 18:53:36.913264 21699 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:36.913451 21699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:36.913808 21699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:36.917235 21699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "971e570555454064967ace6dc9b30846"
format_stamp: "Formatted at 2026-06-19 18:53:36 on dist-test-slave-dm3f"
server_key: "d19ada26eb9881f98b48dde950860d95"
server_key_iv: "9393a1a5526b3c51d98c7d84258538e4"
server_key_version: "encryptionkey@0"
I20260619 18:53:36.918205 21699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "971e570555454064967ace6dc9b30846"
format_stamp: "Formatted at 2026-06-19 18:53:36 on dist-test-slave-dm3f"
server_key: "d19ada26eb9881f98b48dde950860d95"
server_key_iv: "9393a1a5526b3c51d98c7d84258538e4"
server_key_version: "encryptionkey@0"
I20260619 18:53:36.924357 21699 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:53:36.929049 21714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:36.930658 21699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 18:53:36.930940 21699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "971e570555454064967ace6dc9b30846"
format_stamp: "Formatted at 2026-06-19 18:53:36 on dist-test-slave-dm3f"
server_key: "d19ada26eb9881f98b48dde950860d95"
server_key_iv: "9393a1a5526b3c51d98c7d84258538e4"
server_key_version: "encryptionkey@0"
I20260619 18:53:36.931195 21699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:36.942087 21559 heartbeater.cc:503] Master 127.15.91.62:45449 was elected leader, sending a full tablet report...
I20260619 18:53:36.957486 21699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:36.961995 21699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:36.962276 21699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:36.963413 21699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:36.964956 21699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:36.965052 21699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:36.965154 21699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:36.965199 21699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:37.003141 21699 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:33255
I20260619 18:53:37.003211 21826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:33255 every 8 connection(s)
I20260619 18:53:37.004698 21699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:53:37.009956 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 21699
I20260619 18:53:37.010143 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:53:37.010578 15724 external_mini_cluster.cc:1428] Setting key fbb0f00cc1b2abd3a162f7c37aac27bf
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:37.029731 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:37.007922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:35353 (local address 127.15.91.62:45449)
0619 18:53:37.008438 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:37.008442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:37.009516 (+ 1074us) server_negotiation.cc:407] Connection header received
0619 18:53:37.011928 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:37.011934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:37.012043 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:37.012176 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:37.013958 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.014849 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.016678 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.017115 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.017267 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:37.021276 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:37.021282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:37.021285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:37.021324 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:37.024919 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.026187 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.026198 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.026203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.026393 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.027061 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.027068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.027072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.027454 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:37.027657 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:37.028981 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0619 18:53:37.029298 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":270,"threads_started":1}
I20260619 18:53:37.031080 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:37.008472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:45449 (local address 127.15.91.3:35353)
0619 18:53:37.009302 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:37.009346 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:37.011601 (+ 2255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:37.012462 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:37.012472 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:37.013095 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:37.013741 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:37.013764 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.015042 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.015046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:37.016466 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:37.016485 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.017389 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.017396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:37.017657 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:37.018451 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:37.018476 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:37.020966 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:37.025263 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:37.025276 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:37.025305 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:37.026002 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:37.026559 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:37.026564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:37.026567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:37.026927 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:37.027627 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:37.027640 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:37.028402 (+ 762us) client_negotiation.cc:769] Sending connection context
0619 18:53:37.028800 (+ 398us) client_negotiation.cc:240] Negotiation successful
0619 18:53:37.029461 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":321,"threads_started":1}
I20260619 18:53:37.033358 21827 heartbeater.cc:347] Connected to a master server at 127.15.91.62:45449
I20260619 18:53:37.034015 21827 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:37.035084 21827 heartbeater.cc:511] Master 127.15.91.62:45449 requested a full tablet report, sending...
I20260619 18:53:37.037343 21370 ts_manager.cc:194] Registered new tserver with Master: 971e570555454064967ace6dc9b30846 (127.15.91.3:33255)
I20260619 18:53:37.038560 21370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:35353
I20260619 18:53:37.041061 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/test-admin.keytab.
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:37.073632 15724 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 18:53:37.079658 21835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:53:37.076654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.15.91.1:33531)
0619 18:53:37.077377 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:37.077385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:37.077418 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:53:37.077516 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:37.077523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:37.077774 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:37.077914 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:37.079135 (+ 1221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47452: BlockingRecv error: recv got EOF from 127.0.0.1:47452 (error 108)
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":419,"threads_started":1}
W20260619 18:53:37.079761 15724 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:37.081666 15724 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:37.083341 15724 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:53:37.103790 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:37.085854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.15.91.1:33531)
0619 18:53:37.086194 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:37.086203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:37.086649 (+ 446us) server_negotiation.cc:407] Connection header received
0619 18:53:37.086889 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:37.086893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:37.086970 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:37.087073 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:37.088275 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.089330 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.090833 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.091492 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.091620 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:37.096931 (+ 5311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:37.096948 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:37.096966 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:37.097026 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:37.100247 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.101178 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.101188 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.101190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.101356 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.101867 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.101872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.101874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.102297 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:37.102572 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:37.103156 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 18:53:37.103472 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
W20260619 18:53:37.110761 21835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:53:37.108661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.15.91.1:33531)
0619 18:53:37.108937 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:37.108944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:37.109152 (+ 208us) server_negotiation.cc:407] Connection header received
0619 18:53:37.109290 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:37.109295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:37.109375 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:37.109464 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:37.110455 (+ 991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47456: BlockingRecv error: recv got EOF from 127.0.0.1:47456 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260619 18:53:37.110954 15724 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:37.112792 15724 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:37.114423 15724 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:53:37.130860 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:37.116177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.15.91.1:33531)
0619 18:53:37.116401 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:37.116409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:37.116927 (+ 518us) server_negotiation.cc:407] Connection header received
0619 18:53:37.117073 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:37.117077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:37.117149 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:37.117242 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:37.118283 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.119159 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.120454 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.120944 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.121055 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:37.125140 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:37.125146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:37.125150 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:37.125211 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:37.128087 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.128923 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.128928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.128931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.129089 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.129511 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.129516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.129518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.129847 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:37.130077 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:37.130324 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 18:53:37.130517 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 18:53:37.135744 21835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:53:37.133781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.15.91.1:33531)
0619 18:53:37.134022 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:37.134028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:37.134203 (+ 175us) server_negotiation.cc:407] Connection header received
0619 18:53:37.134340 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:37.134344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:37.134424 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:37.134515 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:37.135459 (+ 944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47478: BlockingRecv error: recv got EOF from 127.0.0.1:47478 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260619 18:53:37.135901 15724 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781895164554191-15724-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:37.137688 15724 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:37.139230 15724 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:53:37.156044 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:37.140844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.15.91.1:33531)
0619 18:53:37.141099 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:37.141106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:37.141410 (+ 304us) server_negotiation.cc:407] Connection header received
0619 18:53:37.141647 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:37.141652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:37.141719 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:37.141828 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:37.142922 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.143681 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.145161 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.145693 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.145846 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:37.150031 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:37.150037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:37.150040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:37.150084 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:37.153050 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.153853 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.153859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.153869 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.154020 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.154413 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.154418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.154423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.154788 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:37.155118 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:37.155429 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 18:53:37.155705 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 18:53:37.157810 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 21431
I20260619 18:53:37.170516 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:37.158622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.15.91.1:33531)
0619 18:53:37.158846 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:37.158851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:37.159219 (+ 368us) server_negotiation.cc:407] Connection header received
0619 18:53:37.159361 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:37.159365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:37.159430 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:37.159516 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:37.160463 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.161168 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.162428 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.162949 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.163053 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:37.165285 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:37.165290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:37.165294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:37.165336 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:37.168149 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.168861 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.168866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.168868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.169001 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.169342 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.169345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.169347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.169611 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:37.169826 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:37.170055 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 18:53:37.170241 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:53:37.171233 21554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:37.258111 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 21431
I20260619 18:53:37.275663 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 21565
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:53:37.292744 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:37.276666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.15.91.2:34941)
0619 18:53:37.277276 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:37.277283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:37.277312 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:53:37.277383 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:37.277390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:37.277618 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:37.277767 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:37.278841 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.279907 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.281288 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.281939 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.282069 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:37.286405 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:37.286423 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:37.286440 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:37.286491 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:37.289581 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.290419 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.290427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.290429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.290567 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.291136 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.291140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.291142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.291537 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:37.291788 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:37.292094 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 18:53:37.292404 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":335,"threads_started":1}
I20260619 18:53:37.294090 21688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:37.380549 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 21565
I20260619 18:53:37.400178 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 21699
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:53:37.417662 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:37.401208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.15.91.3:33255)
0619 18:53:37.401894 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:37.401902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:37.401937 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:53:37.401999 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:37.402005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:37.402227 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:37.402428 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:37.403652 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.404817 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.406171 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:37.406952 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:37.407104 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:37.411544 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:37.411561 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:37.411579 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:37.411630 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:37.414902 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.415712 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.415720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.415722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.415860 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:37.416222 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:37.416225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:37.416227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:37.416625 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:37.416886 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:37.417114 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 18:53:37.417346 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":340,"threads_started":1}
I20260619 18:53:37.419142 21822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:37.505110 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 21699
I20260619 18:53:37.522902 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 21338
I20260619 18:53:37.524277 21400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:37.629544 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 21338
2026-06-19T18:53:37Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4223 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21855](info): set up 2 sockets
Jun 19 18:53:37 dist-test-slave-dm3f krb5kdc[21855](info): commencing operation
krb5kdc: starting...
W20260619 18:53:38.743775 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:53:38 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.62.keytab.
Entry for principal oryx/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.62.keytab.
I20260619 18:53:38.970404 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:34269
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.62.keytab
--principal=oryx/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:34269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:39.324239 21868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:39.324640 21868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:39.324819 21868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:39.332638 21868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:39.332830 21868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:39.332953 21868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:39.333055 21868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:39.333143 21868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:39.343819 21868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:34269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.62.keytab
--principal=oryx/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:34269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:39.345458 21868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:39.347033 21868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:39.360316 21873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:39.360307 21874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:39.360383 21876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:39.361444 21868 server_base.cc:1034] running on GCE node
I20260619 18:53:39.362389 21868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:39.364209 21868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:39.365494 21868 hybrid_clock.cc:648] HybridClock initialized: now 1781895219365454 us; error 88 us; skew 500 ppm
I20260619 18:53:39.365924 21868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:39 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895219, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:39.369503 21868 init.cc:377] Logged in from keytab as oryx/127.15.91.62@KRBTEST.COM (short username oryx)
I20260619 18:53:39.371389 21868 webserver.cc:460] Webserver started at http://127.15.91.62:41165/ using document root <none> and password file <none>
I20260619 18:53:39.372272 21868 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:39.372473 21868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:39.372901 21868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:39.376538 21868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "68bcf1108b5f49d398f22fe37bc8c056"
format_stamp: "Formatted at 2026-06-19 18:53:39 on dist-test-slave-dm3f"
server_key: "cdc37503419edec0e1e4bf139b99dd14"
server_key_iv: "6aa75e7a16f047c8f8c52614c027afa0"
server_key_version: "encryptionkey@0"
I20260619 18:53:39.377487 21868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "68bcf1108b5f49d398f22fe37bc8c056"
format_stamp: "Formatted at 2026-06-19 18:53:39 on dist-test-slave-dm3f"
server_key: "cdc37503419edec0e1e4bf139b99dd14"
server_key_iv: "6aa75e7a16f047c8f8c52614c027afa0"
server_key_version: "encryptionkey@0"
I20260619 18:53:39.385131 21868 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260619 18:53:39.390323 21883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:39.392110 21868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:53:39.392355 21868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "68bcf1108b5f49d398f22fe37bc8c056"
format_stamp: "Formatted at 2026-06-19 18:53:39 on dist-test-slave-dm3f"
server_key: "cdc37503419edec0e1e4bf139b99dd14"
server_key_iv: "6aa75e7a16f047c8f8c52614c027afa0"
server_key_version: "encryptionkey@0"
I20260619 18:53:39.392591 21868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:39.407425 21868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:39.412317 21868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:39.412657 21868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:39.433576 21868 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:34269
I20260619 18:53:39.433606 21934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:34269 every 8 connection(s)
I20260619 18:53:39.435240 21868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:39.439509 21935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:39.443460 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 21868
I20260619 18:53:39.443686 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:39.444654 15724 external_mini_cluster.cc:1428] Setting key e7e95f296bb4f4eacbce9539b1b3f73e
I20260619 18:53:39.453925 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056: Bootstrap starting.
I20260619 18:53:39.457548 21935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:39.458788 21935 log.cc:824] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:53:39 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:39.462474 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056: No bootstrap required, opened a new log
I20260619 18:53:39.470109 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:39.447245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.15.91.62:34269)
0619 18:53:39.448155 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:39.448168 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:39.448213 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:53:39.449980 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:39.450011 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:39.450510 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:39.451077 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:39.452754 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:39.454258 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:39.455809 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:39.456599 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:39.456831 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:39.461621 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:39.461631 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:39.461648 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:39.461695 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:39.465134 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:39.466011 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:39.466020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:39.466028 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:39.466239 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:39.466762 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:39.466769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:39.466772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:39.467459 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:39.467866 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:39.468375 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 18:53:39.468905 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":422,"threads_started":1}
I20260619 18:53:39.471414 21935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } }
I20260619 18:53:39.471861 21935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:39.471992 21935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bcf1108b5f49d398f22fe37bc8c056, State: Initialized, Role: FOLLOWER
I20260619 18:53:39.472692 21935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } }
I20260619 18:53:39.472978 21935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:39.473121 21935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:39.473340 21935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:39.476116 21935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } }
I20260619 18:53:39.476882 21935 leader_election.cc:302] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68bcf1108b5f49d398f22fe37bc8c056; no voters:
I20260619 18:53:39.477609 21935 leader_election.cc:288] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:39.477694 21940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:39.479177 21940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 1 LEADER]: Becoming Leader. State: Replica: 68bcf1108b5f49d398f22fe37bc8c056, State: Running, Role: LEADER
I20260619 18:53:39.479799 21940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } }
I20260619 18:53:39.480232 21935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:39.483093 21942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68bcf1108b5f49d398f22fe37bc8c056. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } }
I20260619 18:53:39.483613 21942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:39.483506 21941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } }
I20260619 18:53:39.483848 21941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:39.486155 21949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:39.492372 21949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:39.503031 21949 catalog_manager.cc:1347] Generated new cluster ID: 40b587c88d5540ceb49e34cfd5421bbf
I20260619 18:53:39.503185 21949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:39.526618 21949 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:39.528757 21949 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:39.538190 21949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056: Generated new TSK 0
I20260619 18:53:39.539520 21949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab.
Entry for principal oryx/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab.
I20260619 18:53:39.594949 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:34269
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab
--principal=oryx/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:39.979732 21961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:39.980116 21961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:39.980223 21961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:39.988416 21961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:39.988580 21961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:39.988703 21961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:53:39.999583 21961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab
--principal=oryx/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:40.000980 21961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:40.002410 21961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:40.017982 21966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:40.018083 21967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:40.018054 21969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:40.019513 21961 server_base.cc:1034] running on GCE node
I20260619 18:53:40.020342 21961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:40.021549 21961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:40.022907 21961 hybrid_clock.cc:648] HybridClock initialized: now 1781895220022827 us; error 66 us; skew 500 ppm
I20260619 18:53:40.023368 21961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:40 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895220, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:40.028048 21961 init.cc:377] Logged in from keytab as oryx/127.15.91.1@KRBTEST.COM (short username oryx)
I20260619 18:53:40.029887 21961 webserver.cc:460] Webserver started at http://127.15.91.1:33191/ using document root <none> and password file <none>
I20260619 18:53:40.030829 21961 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:40.031033 21961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:40.031391 21961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:40.034858 21961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "d1f612cc551a4147868957b40b15baf8"
format_stamp: "Formatted at 2026-06-19 18:53:40 on dist-test-slave-dm3f"
server_key: "dd356c2658e9d91ee9ad38463e45fa14"
server_key_iv: "376b1465a356a1567aec9af231337938"
server_key_version: "encryptionkey@0"
I20260619 18:53:40.035883 21961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "d1f612cc551a4147868957b40b15baf8"
format_stamp: "Formatted at 2026-06-19 18:53:40 on dist-test-slave-dm3f"
server_key: "dd356c2658e9d91ee9ad38463e45fa14"
server_key_iv: "376b1465a356a1567aec9af231337938"
server_key_version: "encryptionkey@0"
I20260619 18:53:40.042615 21961 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:53:40.047031 21976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:40.048986 21961 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:53:40.049315 21961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "d1f612cc551a4147868957b40b15baf8"
format_stamp: "Formatted at 2026-06-19 18:53:40 on dist-test-slave-dm3f"
server_key: "dd356c2658e9d91ee9ad38463e45fa14"
server_key_iv: "376b1465a356a1567aec9af231337938"
server_key_version: "encryptionkey@0"
I20260619 18:53:40.049607 21961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:40.073758 21961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:40.078917 21961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:40.079259 21961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:40.080477 21961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:40.082109 21961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:40.082230 21961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:40.082399 21961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:40.082477 21961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:40.120112 21961 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:35739
I20260619 18:53:40.120162 22088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:35739 every 8 connection(s)
I20260619 18:53:40.121613 21961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:53:40.130067 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 21961
I20260619 18:53:40.130297 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:53:40.131057 15724 external_mini_cluster.cc:1428] Setting key f71f460c72c3f334c387126c146fd03e
Jun 19 18:53:40 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895220, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:40.151125 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:40.124701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:57629 (local address 127.15.91.62:34269)
0619 18:53:40.125188 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:40.125193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:40.126485 (+ 1292us) server_negotiation.cc:407] Connection header received
0619 18:53:40.128623 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:40.128630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:40.128737 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:40.128873 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:40.130830 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:40.131659 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:40.134056 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:40.134467 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:40.134577 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:40.143953 (+ 9376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:40.143959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:40.143963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:40.144012 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:40.146935 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:40.148229 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:40.148234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:40.148236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:40.148378 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:40.149020 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:40.149024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:40.149026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:40.149308 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:40.149432 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:40.150534 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0619 18:53:40.150837 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":249,"threads_started":1}
WARNING: no policy specified for oryx/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:53:40.152863 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:40.125368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.1:57629)
0619 18:53:40.126241 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:40.126295 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:40.128259 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:40.129165 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:40.129175 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:40.129769 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:40.130557 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:40.130581 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:40.131874 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:40.131879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:40.133833 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:40.133868 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:40.134790 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:40.134802 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:40.135220 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:40.140581 (+ 5361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:40.140613 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:40.143604 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:40.147169 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:40.147185 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:40.147212 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:40.148042 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:40.148527 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:40.148535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:40.148542 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:40.148874 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:40.149470 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:40.149484 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:40.150048 (+ 564us) client_negotiation.cc:769] Sending connection context
0619 18:53:40.150439 (+ 391us) client_negotiation.cc:240] Negotiation successful
0619 18:53:40.151149 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":342,"threads_started":1}
I20260619 18:53:40.154994 22089 heartbeater.cc:347] Connected to a master server at 127.15.91.62:34269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.2@KRBTEST.COM" created.
I20260619 18:53:40.155445 22089 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:40.156551 22089 heartbeater.cc:511] Master 127.15.91.62:34269 requested a full tablet report, sending...
I20260619 18:53:40.159359 21900 ts_manager.cc:194] Registered new tserver with Master: d1f612cc551a4147868957b40b15baf8 (127.15.91.1:35739)
I20260619 18:53:40.160990 21900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:57629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab.
Entry for principal oryx/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab.
I20260619 18:53:40.183357 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:34269
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab
--principal=oryx/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:40.549330 22095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:40.549670 22095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:40.549770 22095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:40.557840 22095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:40.557986 22095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:40.558132 22095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:53:40.568802 22095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab
--principal=oryx/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:40.570202 22095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:40.571692 22095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:40.588526 22100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:40.588833 22103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:40.588907 22101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:40.589761 22095 server_base.cc:1034] running on GCE node
I20260619 18:53:40.590576 22095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:40.591797 22095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:40.593116 22095 hybrid_clock.cc:648] HybridClock initialized: now 1781895220593061 us; error 77 us; skew 500 ppm
I20260619 18:53:40.593626 22095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:40 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895220, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:40.597998 22095 init.cc:377] Logged in from keytab as oryx/127.15.91.2@KRBTEST.COM (short username oryx)
I20260619 18:53:40.599931 22095 webserver.cc:460] Webserver started at http://127.15.91.2:41867/ using document root <none> and password file <none>
I20260619 18:53:40.600844 22095 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:40.601034 22095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:40.601418 22095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:40.605078 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "ebb4667c5c934fd090783186dd64a3a2"
format_stamp: "Formatted at 2026-06-19 18:53:40 on dist-test-slave-dm3f"
server_key: "6c17d41701e635c127468468b82ffd73"
server_key_iv: "79910c692e01b44a5aa4508f9aacdfdf"
server_key_version: "encryptionkey@0"
I20260619 18:53:40.606223 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "ebb4667c5c934fd090783186dd64a3a2"
format_stamp: "Formatted at 2026-06-19 18:53:40 on dist-test-slave-dm3f"
server_key: "6c17d41701e635c127468468b82ffd73"
server_key_iv: "79910c692e01b44a5aa4508f9aacdfdf"
server_key_version: "encryptionkey@0"
I20260619 18:53:40.612906 22095 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260619 18:53:40.617295 22110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:40.619079 22095 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:53:40.619315 22095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "ebb4667c5c934fd090783186dd64a3a2"
format_stamp: "Formatted at 2026-06-19 18:53:40 on dist-test-slave-dm3f"
server_key: "6c17d41701e635c127468468b82ffd73"
server_key_iv: "79910c692e01b44a5aa4508f9aacdfdf"
server_key_version: "encryptionkey@0"
I20260619 18:53:40.619547 22095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:40.640239 22095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:40.645368 22095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:40.645725 22095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:40.646975 22095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:40.648579 22095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:40.648669 22095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:40.648838 22095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:40.648890 22095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:40.685809 22095 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:39293
I20260619 18:53:40.685858 22222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:39293 every 8 connection(s)
I20260619 18:53:40.687384 22095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:53:40.687966 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 22095
I20260619 18:53:40.688191 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:53:40.688879 15724 external_mini_cluster.cc:1428] Setting key 463dfe3d2bcc1feb0d6cae429205d759
Jun 19 18:53:40 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895220, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.3@KRBTEST.COM" created.
I20260619 18:53:40.716718 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:40.691065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:60251 (local address 127.15.91.62:34269)
0619 18:53:40.691605 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:40.691612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:40.693007 (+ 1395us) server_negotiation.cc:407] Connection header received
0619 18:53:40.696347 (+ 3340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:40.696352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:40.696479 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:40.696663 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:40.698803 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:40.699557 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:40.702615 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:40.703174 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:40.703319 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:40.708720 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:40.708732 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:40.708739 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:40.708795 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:40.712816 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:40.713782 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:40.713790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:40.713794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:40.713966 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:40.714512 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:40.714518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:40.714522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:40.714955 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:40.715119 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:40.716001 (+ 882us) server_negotiation.cc:299] Negotiation successful
0619 18:53:40.716277 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":275,"threads_started":1}
I20260619 18:53:40.717947 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:40.691760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.2:60251)
0619 18:53:40.692772 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:40.692836 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:40.695842 (+ 3006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:40.697032 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:40.697043 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:40.697824 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:40.698523 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:40.698546 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:40.699774 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:40.699783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:40.702291 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:40.702321 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:40.703546 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:40.703558 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:40.703989 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:40.704945 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:40.704991 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:40.708194 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:40.713044 (+ 4850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:40.713056 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:40.713085 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:40.713612 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:40.714118 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:40.714124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:40.714127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:40.714367 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:40.715114 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:40.715125 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:40.715548 (+ 423us) client_negotiation.cc:769] Sending connection context
0619 18:53:40.715975 (+ 427us) client_negotiation.cc:240] Negotiation successful
0619 18:53:40.716703 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":367,"threads_started":1}
I20260619 18:53:40.719797 22223 heartbeater.cc:347] Connected to a master server at 127.15.91.62:34269
I20260619 18:53:40.720208 22223 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:40.721274 22223 heartbeater.cc:511] Master 127.15.91.62:34269 requested a full tablet report, sending...
I20260619 18:53:40.723750 21900 ts_manager.cc:194] Registered new tserver with Master: ebb4667c5c934fd090783186dd64a3a2 (127.15.91.2:39293)
I20260619 18:53:40.724902 21900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:60251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab.
Entry for principal oryx/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab.
I20260619 18:53:40.747804 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:34269
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab
--principal=oryx/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:41.112223 22229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:41.112541 22229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:41.112638 22229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:41.120618 22229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:41.120743 22229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:41.120949 22229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:53:41.132114 22229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab
--principal=oryx/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:41.133582 22229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:41.135154 22229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:41.150997 22237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:41.151077 22235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:41.151422 22234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:41.156615 22229 server_base.cc:1034] running on GCE node
I20260619 18:53:41.157388 22229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:41.158561 22229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:41.159835 22229 hybrid_clock.cc:648] HybridClock initialized: now 1781895221159765 us; error 93 us; skew 500 ppm
I20260619 18:53:41.160286 22229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:41 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895221, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:41.164170 22229 init.cc:377] Logged in from keytab as oryx/127.15.91.3@KRBTEST.COM (short username oryx)
I20260619 18:53:41.164651 22089 heartbeater.cc:503] Master 127.15.91.62:34269 was elected leader, sending a full tablet report...
I20260619 18:53:41.166460 22229 webserver.cc:460] Webserver started at http://127.15.91.3:33889/ using document root <none> and password file <none>
I20260619 18:53:41.167408 22229 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:41.167537 22229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:41.167807 22229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:41.171348 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "a60a1830734746bfa8ab432167bf9cef"
format_stamp: "Formatted at 2026-06-19 18:53:41 on dist-test-slave-dm3f"
server_key: "ee4b9b7b1d7289fff9760da3d802f512"
server_key_iv: "a030c991d1fba318ba84cf6127696ca6"
server_key_version: "encryptionkey@0"
I20260619 18:53:41.172319 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "a60a1830734746bfa8ab432167bf9cef"
format_stamp: "Formatted at 2026-06-19 18:53:41 on dist-test-slave-dm3f"
server_key: "ee4b9b7b1d7289fff9760da3d802f512"
server_key_iv: "a030c991d1fba318ba84cf6127696ca6"
server_key_version: "encryptionkey@0"
I20260619 18:53:41.178529 22229 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 18:53:41.183107 22244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:41.185007 22229 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:53:41.185199 22229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "a60a1830734746bfa8ab432167bf9cef"
format_stamp: "Formatted at 2026-06-19 18:53:41 on dist-test-slave-dm3f"
server_key: "ee4b9b7b1d7289fff9760da3d802f512"
server_key_iv: "a030c991d1fba318ba84cf6127696ca6"
server_key_version: "encryptionkey@0"
I20260619 18:53:41.185421 22229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:41.207072 22229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:41.212271 22229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:41.212558 22229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:41.213721 22229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:41.215842 22229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:41.215948 22229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:41.216053 22229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:41.216094 22229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:41.254524 22229 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:38517
I20260619 18:53:41.254535 22356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:38517 every 8 connection(s)
I20260619 18:53:41.256135 22229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:53:41.262439 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 22229
I20260619 18:53:41.262657 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:53:41.263195 15724 external_mini_cluster.cc:1428] Setting key c461b1513758a3d5d35c2789f228df38
Jun 19 18:53:41 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895221, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:41.281894 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.259940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:44239 (local address 127.15.91.62:34269)
0619 18:53:41.260458 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:41.260463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:41.261718 (+ 1255us) server_negotiation.cc:407] Connection header received
0619 18:53:41.264354 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:41.264358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:41.264432 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:41.264580 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:41.266685 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.267624 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.269427 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.269872 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.269987 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.274552 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:41.274558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:41.274562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:41.274605 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:41.277773 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.278898 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.278903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.278905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.279132 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.279795 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.279803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.279807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.280278 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:41.280413 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:41.281344 (+ 931us) server_negotiation.cc:299] Negotiation successful
0619 18:53:41.281571 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":259,"threads_started":1}
I20260619 18:53:41.283123 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.260650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.3:44239)
0619 18:53:41.261509 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:41.261558 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:41.263692 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:41.264982 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:41.264999 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:41.265732 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:41.266438 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.266481 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.267812 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.267817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.269229 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.269246 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.270118 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.270123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.270359 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.271231 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:41.271258 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:41.274211 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:41.277997 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.278010 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.278038 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.278718 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.279303 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.279310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.279313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.279643 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.280445 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:41.280453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:41.280882 (+ 429us) client_negotiation.cc:769] Sending connection context
0619 18:53:41.281152 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 18:53:41.281743 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":343,"threads_started":1}
I20260619 18:53:41.284991 22357 heartbeater.cc:347] Connected to a master server at 127.15.91.62:34269
I20260619 18:53:41.285422 22357 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:41.286347 22357 heartbeater.cc:511] Master 127.15.91.62:34269 requested a full tablet report, sending...
I20260619 18:53:41.288362 21900 ts_manager.cc:194] Registered new tserver with Master: a60a1830734746bfa8ab432167bf9cef (127.15.91.3:38517)
I20260619 18:53:41.289341 21900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:44239
I20260619 18:53:41.293130 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.91.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.61.keytab.
Entry for principal oryx/127.15.91.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.61.keytab.
I20260619 18:53:41.344833 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.61:35667
--webserver_interface=127.15.91.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.61.keytab
--principal=oryx/127.15.91.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:34269,127.15.91.61:35667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:41.712877 22363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:41.713361 22363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:41.713559 22363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:41.722323 22363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:41.722527 22363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:41.722627 22363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:41.722739 22363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:41.722810 22363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:41.728108 22223 heartbeater.cc:503] Master 127.15.91.62:34269 was elected leader, sending a full tablet report...
I20260619 18:53:41.734104 22363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:34269,127.15.91.61:35667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.61.keytab
--principal=oryx/127.15.91.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.61:35667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.91.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
W20260619 18:53:41.734598 22363 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.91.62:34269,127.15.91.61:35667'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 18:53:41.735785 22363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:41.737268 22363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:41.750692 22369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:41.750667 22368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:41.750630 22371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:41.751751 22363 server_base.cc:1034] running on GCE node
I20260619 18:53:41.752621 22363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:41.754407 22363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:41.755685 22363 hybrid_clock.cc:648] HybridClock initialized: now 1781895221755586 us; error 76 us; skew 500 ppm
I20260619 18:53:41.756083 22363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:41 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895221, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:41.759727 22363 init.cc:377] Logged in from keytab as oryx/127.15.91.61@KRBTEST.COM (short username oryx)
I20260619 18:53:41.761605 22363 webserver.cc:460] Webserver started at http://127.15.91.61:34941/ using document root <none> and password file <none>
I20260619 18:53:41.762460 22363 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:41.762610 22363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:41.763015 22363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:41.766455 22363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data/instance:
uuid: "3943f18ae29c4d5a99391d6b77d60a8e"
format_stamp: "Formatted at 2026-06-19 18:53:41 on dist-test-slave-dm3f"
server_key: "0ea1b1834f76f5171dba2bb0fbe974dc"
server_key_iv: "7e01566e10443243f1d4914f8f61f14e"
server_key_version: "encryptionkey@0"
I20260619 18:53:41.767575 22363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/wal/instance:
uuid: "3943f18ae29c4d5a99391d6b77d60a8e"
format_stamp: "Formatted at 2026-06-19 18:53:41 on dist-test-slave-dm3f"
server_key: "0ea1b1834f76f5171dba2bb0fbe974dc"
server_key_iv: "7e01566e10443243f1d4914f8f61f14e"
server_key_version: "encryptionkey@0"
I20260619 18:53:41.774015 22363 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:53:41.778539 22378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:41.780292 22363 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:53:41.780643 22363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/wal
uuid: "3943f18ae29c4d5a99391d6b77d60a8e"
format_stamp: "Formatted at 2026-06-19 18:53:41 on dist-test-slave-dm3f"
server_key: "0ea1b1834f76f5171dba2bb0fbe974dc"
server_key_iv: "7e01566e10443243f1d4914f8f61f14e"
server_key_version: "encryptionkey@0"
I20260619 18:53:41.780922 22363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:41.808214 22363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:41.813302 22363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:41.813668 22363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:41.835325 22363 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.61:35667
I20260619 18:53:41.835347 22429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.61:35667 every 8 connection(s)
I20260619 18:53:41.836985 22363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
I20260619 18:53:41.838397 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 22363
I20260619 18:53:41.842622 22430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:41.852043 22430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:53:41 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.91.61@KRBTEST.COM
Jun 19 18:53:41 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895221, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.61@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:41.866653 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.841071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.15.91.61:35667)
0619 18:53:41.842176 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:41.842195 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:41.842563 (+ 368us) server_negotiation.cc:407] Connection header received
0619 18:53:41.844524 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:41.844566 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:41.845030 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:41.845440 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:41.847518 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.849034 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.850993 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.851760 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.851939 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.858013 (+ 6074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:41.858024 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:41.858036 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:41.858086 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:41.861381 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.862314 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.862322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.862330 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.862546 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.863165 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.863170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.863176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.864088 (+ 912us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:41.864456 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:41.864981 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 18:53:41.865485 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":464,"threads_started":1}
I20260619 18:53:41.872285 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.855903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:44554)
0619 18:53:41.856925 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:41.856978 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:41.857170 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:41.857891 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:41.857899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:41.858647 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:41.859050 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.859062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.860169 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.860174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.861395 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.861412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.862266 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.862274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.862476 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.863359 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:41.863381 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:41.866184 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:41.869636 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.869646 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.869652 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.870156 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.870543 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.870547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.870549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.870693 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.871319 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:41.871326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:41.871498 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 18:53:41.871670 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 18:53:41.871968 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"spinlock_wait_cycles":20864,"thread_start_us":391,"threads_started":1}
I20260619 18:53:41.872298 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.855933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.15.91.62:34269)
0619 18:53:41.857493 (+ 1560us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:41.857497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:41.857514 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:53:41.857578 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:41.857581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:41.857642 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:41.857779 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:41.859248 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.859987 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.861585 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.862024 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.862189 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.866434 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:41.866440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:41.866444 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:41.866486 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:41.869450 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.870289 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.870293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.870295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.870434 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.870840 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.870843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.870846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.871201 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:41.871321 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:41.871854 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 18:53:41.872058 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1441,"thread_start_us":1332,"threads_started":1}
I20260619 18:53:41.874262 22430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:53:41 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895221, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.61@KRBTEST.COM for oryx/127.15.91.61@KRBTEST.COM
I20260619 18:53:41.887969 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.875279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35667 (local address 127.0.0.1:33748)
0619 18:53:41.875569 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:41.875585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:41.875684 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:41.875958 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:41.875961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:41.876342 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:41.876484 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.876492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.877477 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.877480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.878363 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.878374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.879178 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.879182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.879333 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.879986 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:41.880005 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:41.882173 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:41.885378 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.885383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.885386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.885890 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.886286 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.886291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.886294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.886446 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.887143 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:41.887146 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:41.887313 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 18:53:41.887433 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 18:53:41.887709 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260619 18:53:41.888347 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.875286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.15.91.61:35667)
0619 18:53:41.875483 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:41.875488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:41.875650 (+ 162us) server_negotiation.cc:407] Connection header received
0619 18:53:41.875772 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:41.875776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:41.875836 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:41.875930 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:41.876612 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.877339 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.878506 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.879035 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.879138 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.882375 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:41.882380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:41.882383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:41.882418 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:41.885191 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.886025 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.886029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.886031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.886173 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.886612 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.886620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.886625 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.887035 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:41.887162 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:41.887542 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 18:53:41.887877 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":81}
I20260619 18:53:41.893585 22430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: Bootstrap starting.
I20260619 18:53:41.897171 22430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:41.898412 22430 log.cc:824] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:41.901557 22430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: No bootstrap required, opened a new log
I20260619 18:53:41.909109 22430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } }
I20260619 18:53:41.909541 22430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:41.909659 22430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3943f18ae29c4d5a99391d6b77d60a8e, State: Initialized, Role: FOLLOWER
I20260619 18:53:41.910310 22430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } }
I20260619 18:53:41.911324 22437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } } }
I20260619 18:53:41.911645 22437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:53:41.912509 22430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:53:41.922662 22448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:53:41.922865 22448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:53:41.925115 22363 master_runner.cc:428] Detected that this master 3943f18ae29c4d5a99391d6b77d60a8e is joining an existing cluster
I20260619 18:53:41.925252 22363 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.91.61:35667
I20260619 18:53:41.932960 22437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:41.933378 22437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } }
I20260619 18:53:41.935405 22437 leader_election.cc:288] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68bcf1108b5f49d398f22fe37bc8c056 (127.15.91.62:34269)
I20260619 18:53:41.936780 21910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68bcf1108b5f49d398f22fe37bc8c056" is_pre_election: true
I20260619 18:53:41.938566 22382 leader_election.cc:302] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 3943f18ae29c4d5a99391d6b77d60a8e; no voters: 68bcf1108b5f49d398f22fe37bc8c056
I20260619 18:53:41.939635 22437 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
WARNING: no policy specified for oryx/127.15.91.60@KRBTEST.COM; defaulting to no policy
I20260619 18:53:41.962836 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.930272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.15.91.62:34269)
0619 18:53:41.930482 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:41.930489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:41.944320 (+ 13831us) server_negotiation.cc:407] Connection header received
0619 18:53:41.944552 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:41.944558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:41.944657 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:41.944748 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:41.946892 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.948010 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.950417 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.951193 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.951328 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.954786 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:41.954795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:41.954800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:41.954845 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:41.959087 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.960209 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.960218 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.960224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.960408 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.960895 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.960901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.960906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.961309 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:41.961441 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:41.962094 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 18:53:41.962397 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:53:41.963505 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.930275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.15.91.61:35667)
0619 18:53:41.930693 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:41.930698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:41.943487 (+ 12789us) server_negotiation.cc:407] Connection header received
0619 18:53:41.944580 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:41.944585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:41.944652 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:41.944730 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:41.946558 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.947850 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.950379 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.951224 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.951348 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.954864 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:41.954873 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:41.954880 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:41.954933 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:41.959194 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.960281 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.960287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.960290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.960454 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.960975 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.960980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.960984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.961356 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:41.961520 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:41.962072 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 18:53:41.962511 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":213,"server-negotiator.queue_time_us":72}
I20260619 18:53:41.964063 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.930327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35667 (local address 127.0.0.1:33760)
0619 18:53:41.943331 (+ 13004us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:41.943361 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:41.943577 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:41.944932 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:41.944935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:41.945891 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:41.946349 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.946369 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.948098 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.948106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.950201 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.950222 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.951746 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.951755 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.952122 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.952942 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:41.952974 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:41.954578 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:41.959388 (+ 4810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.959393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.959396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.960133 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.960612 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.960616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.960620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.960808 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.961606 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:41.961610 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:41.961794 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 18:53:41.961986 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 18:53:41.962358 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12838,"mutex_wait_us":86,"thread_start_us":12670,"threads_started":1}
I20260619 18:53:41.964690 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.931139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:44566)
0619 18:53:41.944211 (+ 13072us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:41.944230 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:41.944420 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:41.944926 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:41.944931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:41.946231 (+ 1300us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:41.946635 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.946650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.948257 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.948262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.950180 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.950206 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.951706 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.951717 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.952025 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.953087 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:41.953111 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:41.954431 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:41.959345 (+ 4914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.959351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.959356 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.960006 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.960574 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.960579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.960583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.960754 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.961546 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:41.961552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:41.961776 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 18:53:41.963201 (+ 1425us) client_negotiation.cc:240] Negotiation successful
0619 18:53:41.964151 (+ 950us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12941,"mutex_wait_us":101,"thread_start_us":12700,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.60@KRBTEST.COM" created.
Jun 19 18:53:41 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895219, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.62@KRBTEST.COM for oryx/127.15.91.61@KRBTEST.COM
I20260619 18:53:41.994110 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.971237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35667 (local address 127.0.0.1:33772)
0619 18:53:41.972067 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:41.972116 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:41.972379 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:41.973129 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:41.973139 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:41.974075 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:41.974395 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.974410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.976296 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.976305 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.978360 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:41.978388 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.979703 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.979712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:41.979989 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.980904 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:41.980935 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:41.984532 (+ 3597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:41.990502 (+ 5970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.990525 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.990535 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.991305 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.991889 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:41.991897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:41.991906 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:41.992076 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:41.992789 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:41.992800 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:41.993065 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 18:53:41.993348 (+ 283us) client_negotiation.cc:240] Negotiation successful
0619 18:53:41.993735 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":427,"threads_started":1}
I20260619 18:53:41.994234 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:41.971250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.15.91.61:35667)
0619 18:53:41.971506 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:41.971517 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:41.972216 (+ 699us) server_negotiation.cc:407] Connection header received
0619 18:53:41.972628 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:41.972637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:41.972767 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:41.972840 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:41.974647 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.976052 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.978647 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:41.979449 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:41.979630 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:41.984820 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:41.984830 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:41.984838 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:41.984899 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:41.990198 (+ 5299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.991487 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.991492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.991495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.991725 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:41.992211 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:41.992217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:41.992220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:41.992616 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:41.992793 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:41.993495 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 18:53:41.993774 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:53:41.996997 21900 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.91.61:35667: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e"
member_type: NON_VOTER
last_known_addr {
host: "127.15.91.61"
port: 35667
}
attrs {
promote: true
}
}
dest_uuid: "68bcf1108b5f49d398f22fe37bc8c056"
cas_config_opid_index: -1
I20260619 18:53:41.998287 21900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.60.keytab.
Entry for principal oryx/127.15.91.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.60.keytab.
I20260619 18:53:42.001313 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.60:43887
--webserver_interface=127.15.91.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.60.keytab
--principal=oryx/127.15.91.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:34269,127.15.91.61:35667,127.15.91.60:43887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:53:42.008307 22460 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3943f18ae29c4d5a99391d6b77d60a8e (127.15.91.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } } }
I20260619 18:53:42.009447 22405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:42.013605 22405 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 FOLLOWER]: Refusing update from remote peer 68bcf1108b5f49d398f22fe37bc8c056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 18:53:42.013968 22460 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.91.61:35667
I20260619 18:53:42.014541 22461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } } }
I20260619 18:53:42.015029 22461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:42.020800 22460 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:42.026676 22363 master.cc:537] Master@127.15.91.61:35667 shutting down...
W20260619 18:53:42.031802 21884 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 18:53:42.043248 22363 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:42.043814 22363 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:42.044023 22363 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: stopping tablet replica
W20260619 18:53:42.048130 22380 connection.cc:441] server connection from 127.0.0.1:33772 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33772 (request call id 3) could send its response
W20260619 18:53:42.048945 21884 consensus_peers.cc:487] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 -> Peer 3943f18ae29c4d5a99391d6b77d60a8e (127.15.91.61:35667): Couldn't send request to peer 3943f18ae29c4d5a99391d6b77d60a8e. Status: Network error: recv got EOF from 127.15.91.61:35667 (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260619 18:53:42.052408 22363 master.cc:552] Master@127.15.91.61:35667 shutdown complete.
I20260619 18:53:42.052531 22363 master_runner.cc:305] Clearing existing system tablet
I20260619 18:53:42.052990 22363 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:42.053112 22363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:42.056520 22363 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.005s sys 0.000s
I20260619 18:53:42.058482 22467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:42.058996 22363 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:42.059293 22363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:53:42.059386 22363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/wal
uuid: "3943f18ae29c4d5a99391d6b77d60a8e"
format_stamp: "Formatted at 2026-06-19 18:53:41 on dist-test-slave-dm3f"
server_key: "0ea1b1834f76f5171dba2bb0fbe974dc"
server_key_iv: "7e01566e10443243f1d4914f8f61f14e"
server_key_version: "encryptionkey@0"
I20260619 18:53:42.064603 22363 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:42.072399 22363 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 18:53:42.072638 22363 log.cc:1196] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 18:53:42.073112 22363 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: Deleting consensus metadata
I20260619 18:53:42.073408 22363 master_runner.cc:315] Copying system tablet from 127.15.91.62:34269
I20260619 18:53:42.076412 22363 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.91.62:34269
I20260619 18:53:42.095793 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.077407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44582 (local address 127.15.91.62:34269)
0619 18:53:42.077627 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.077633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.079419 (+ 1786us) server_negotiation.cc:407] Connection header received
0619 18:53:42.079661 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.079666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.079757 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.079888 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:42.081056 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.082256 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.084530 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.085249 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.085422 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.088082 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:42.088090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:42.088094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:42.088143 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:42.092128 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.093200 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.093207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.093210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.093447 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.093973 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.093977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.093981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.094384 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:42.094554 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.095144 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.095413 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:53:42.095902 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.077345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:44582)
0619 18:53:42.079222 (+ 1877us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:42.079252 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:42.079480 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:42.079944 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:42.079950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:42.080533 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:42.080837 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.080852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.082489 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.082496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.084246 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.084268 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.085484 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.085491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.085701 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.086503 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:42.086539 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:42.087758 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:42.092363 (+ 4605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.092371 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.092378 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.092997 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.093597 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.093603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.093607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.093816 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.094545 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:42.094551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:42.094804 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 18:53:42.094982 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 18:53:42.095388 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1700,"thread_start_us":1578,"threads_started":1}
I20260619 18:53:42.097289 21920 tablet_copy_service.cc:140] P 68bcf1108b5f49d398f22fe37bc8c056: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3943f18ae29c4d5a99391d6b77d60a8e ({username='oryx', principal='oryx/127.15.91.61@KRBTEST.COM'} at 127.0.0.1:44582)
I20260619 18:53:42.097632 21920 tablet_copy_service.cc:161] P 68bcf1108b5f49d398f22fe37bc8c056: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3943f18ae29c4d5a99391d6b77d60a8e at {username='oryx', principal='oryx/127.15.91.61@KRBTEST.COM'} at 127.0.0.1:44582: session id = 3943f18ae29c4d5a99391d6b77d60a8e-00000000000000000000000000000000
I20260619 18:53:42.102836 21920 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 18:53:42.105798 22363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 18:53:42.113763 22363 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: tablet copy: Starting download of 0 data blocks...
I20260619 18:53:42.114171 22363 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: tablet copy: Starting download of 1 WAL segments...
I20260619 18:53:42.117750 22363 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 18:53:42.123366 21920 tablet_copy_service.cc:342] P 68bcf1108b5f49d398f22fe37bc8c056: Request end of tablet copy session 3943f18ae29c4d5a99391d6b77d60a8e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.91.61@KRBTEST.COM'} at 127.0.0.1:44582
I20260619 18:53:42.123723 21920 tablet_copy_service.cc:433] P 68bcf1108b5f49d398f22fe37bc8c056: ending tablet copy session 3943f18ae29c4d5a99391d6b77d60a8e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3943f18ae29c4d5a99391d6b77d60a8e
I20260619 18:53:42.139398 22363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:42.142647 22474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:42.144222 22475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:42.152655 22477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:42.156471 22363 server_base.cc:1034] running on GCE node
I20260619 18:53:42.157006 22363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:42.157856 22363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:42.159176 22363 hybrid_clock.cc:648] HybridClock initialized: now 1781895222159106 us; error 45 us; skew 500 ppm
I20260619 18:53:42.159448 22363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:42 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895222, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:42.161803 22363 init.cc:377] Logged in from keytab as oryx/127.15.91.61@KRBTEST.COM (short username oryx)
I20260619 18:53:42.169458 22363 webserver.cc:460] Webserver started at http://127.15.91.61:45987/ using document root <none> and password file <none>
I20260619 18:53:42.170037 22363 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:42.170168 22363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:42.174434 22363 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:53:42.176970 22484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:42.177511 22363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:53:42.177762 22363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/wal
uuid: "3943f18ae29c4d5a99391d6b77d60a8e"
format_stamp: "Formatted at 2026-06-19 18:53:41 on dist-test-slave-dm3f"
server_key: "0ea1b1834f76f5171dba2bb0fbe974dc"
server_key_iv: "7e01566e10443243f1d4914f8f61f14e"
server_key_version: "encryptionkey@0"
I20260619 18:53:42.178020 22363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:42.198680 22363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:42.203325 22363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:42.235123 22363 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.61:35667
I20260619 18:53:42.235180 22535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.61:35667 every 8 connection(s)
I20260619 18:53:42.236657 22363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
I20260619 18:53:42.243243 22536 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:53:42.247905 22536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: Bootstrap starting.
I20260619 18:53:42.276439 22536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:53:42.277045 22536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: Bootstrap complete.
I20260619 18:53:42.278270 22536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } }
I20260619 18:53:42.278605 22536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3943f18ae29c4d5a99391d6b77d60a8e, State: Initialized, Role: LEARNER
I20260619 18:53:42.278940 22536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } }
I20260619 18:53:42.280045 22539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } } }
I20260619 18:53:42.280483 22539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: This master's current role is: LEARNER
I20260619 18:53:42.280493 22536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:42.293061 22357 heartbeater.cc:503] Master 127.15.91.62:34269 was elected leader, sending a full tablet report...
I20260619 18:53:42.294402 22550 catalog_manager.cc:1259] Loaded cluster ID: 40b587c88d5540ceb49e34cfd5421bbf
I20260619 18:53:42.294623 22550 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: loading cluster ID for follower catalog manager: success
I20260619 18:53:42.298703 22550 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: acquiring CA information for follower catalog manager: success
I20260619 18:53:42.302179 22550 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 18:53:42.461144 22462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:42.461472 22462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:42.461578 22462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:42.469411 22462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:42.469558 22462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:42.469606 22462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:42.469658 22462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:42.469703 22462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:42.480703 22462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:34269,127.15.91.61:35667,127.15.91.60:43887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.60.keytab
--principal=oryx/127.15.91.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.60:43887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.91.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:42.482226 22462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:42.483877 22462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:42.497864 22555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:42.497912 22556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:42.498185 22558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:42.504545 22462 server_base.cc:1034] running on GCE node
I20260619 18:53:42.505566 22462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:42.507686 22462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:42.508980 22462 hybrid_clock.cc:648] HybridClock initialized: now 1781895222508898 us; error 86 us; skew 500 ppm
I20260619 18:53:42.509479 22462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:42 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895222, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:42.512926 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.506089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35667 (local address 127.0.0.1:33776)
0619 18:53:42.506750 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:42.506773 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:42.506932 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:42.507385 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:42.507393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:42.507427 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:42.507648 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.507658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.509159 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.509167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.510566 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.510586 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.511943 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.511950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.512164 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.512253 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 18:53:42.512383 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 18:53:42.512562 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":287,"threads_started":1}
I20260619 18:53:42.512926 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.506299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.15.91.61:35667)
0619 18:53:42.507020 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.507028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.507047 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:53:42.507125 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.507129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.507205 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.507340 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:42.507809 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.508974 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.510795 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.511711 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.511896 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.512176 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.512415 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.512566 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":419,"threads_started":1}
I20260619 18:53:42.513243 22462 init.cc:377] Logged in from keytab as oryx/127.15.91.60@KRBTEST.COM (short username oryx)
I20260619 18:53:42.514395 22511 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 18:53:42.515148 22539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68bcf1108b5f49d398f22fe37bc8c056. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } } }
I20260619 18:53:42.515463 22539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: This master's current role is: LEARNER
I20260619 18:53:42.515570 22462 webserver.cc:460] Webserver started at http://127.15.91.60:38875/ using document root <none> and password file <none>
I20260619 18:53:42.516434 22462 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:42.516563 22462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:42.516932 22462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:42.520530 22462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data/instance:
uuid: "ccea9e7e30e34087952516eb8a4ac28b"
format_stamp: "Formatted at 2026-06-19 18:53:42 on dist-test-slave-dm3f"
server_key: "ccb39b53205645b47f6672d1375e4cfb"
server_key_iv: "18f00770f4b80c55d9621590c5328831"
server_key_version: "encryptionkey@0"
I20260619 18:53:42.521447 22462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/wal/instance:
uuid: "ccea9e7e30e34087952516eb8a4ac28b"
format_stamp: "Formatted at 2026-06-19 18:53:42 on dist-test-slave-dm3f"
server_key: "ccb39b53205645b47f6672d1375e4cfb"
server_key_iv: "18f00770f4b80c55d9621590c5328831"
server_key_version: "encryptionkey@0"
I20260619 18:53:42.528121 22462 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:53:42.532929 22567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:42.534595 22462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:53:42.534888 22462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/wal
uuid: "ccea9e7e30e34087952516eb8a4ac28b"
format_stamp: "Formatted at 2026-06-19 18:53:42 on dist-test-slave-dm3f"
server_key: "ccb39b53205645b47f6672d1375e4cfb"
server_key_iv: "18f00770f4b80c55d9621590c5328831"
server_key_version: "encryptionkey@0"
I20260619 18:53:42.535159 22462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:42.553895 22462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:42.558758 22462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:42.559109 22462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:42.581398 22462 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.60:43887
I20260619 18:53:42.581431 22618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.60:43887 every 8 connection(s)
I20260619 18:53:42.583442 22462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
I20260619 18:53:42.588388 22619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:42.590112 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 22462
I20260619 18:53:42.600402 22619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:53:42 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.91.60@KRBTEST.COM
Jun 19 18:53:42 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895222, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.60@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:42.618952 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.593153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.15.91.60:43887)
0619 18:53:42.594219 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.594234 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.594290 (+ 56us) server_negotiation.cc:407] Connection header received
0619 18:53:42.596386 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.596420 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.597042 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.597503 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:42.599615 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.601144 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.603201 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.604610 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.604863 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.609649 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:42.609658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:42.609672 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:42.609719 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:42.613660 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.614508 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.614515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.614524 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.614697 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.615290 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.615295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.615297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.616092 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:42.616462 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.617184 (+ 722us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.617677 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":495,"threads_started":1}
I20260619 18:53:42.621042 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.603991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.15.91.62:34269)
0619 18:53:42.604561 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.604568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.605190 (+ 622us) server_negotiation.cc:407] Connection header received
0619 18:53:42.605446 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.605450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.605520 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.605753 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:42.607104 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.607883 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.609485 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.609918 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.610091 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.614196 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:42.614201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:42.614205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:42.614243 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:42.617714 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.618588 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.618596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.618604 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.618823 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.619338 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.619345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.619349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.619661 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:42.619843 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.620364 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.620640 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":277,"threads_started":1}
I20260619 18:53:42.621121 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.604011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:44584)
0619 18:53:42.605013 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:42.605073 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:42.605269 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:42.605831 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:42.605840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:42.606558 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:42.606914 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.606926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.608066 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.608072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.609290 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.609313 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.610136 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.610149 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.610350 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.610996 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:42.611017 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:42.613903 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:42.617883 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.617896 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.617905 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.618425 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.618954 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.618960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.618964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.619184 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.619820 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:42.619828 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:42.620035 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 18:53:42.620251 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 18:53:42.620640 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":506,"threads_started":1}
I20260619 18:53:42.622606 22619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:53:42 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895222, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.60@KRBTEST.COM for oryx/127.15.91.61@KRBTEST.COM
I20260619 18:53:42.636739 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.623533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35667 (local address 127.0.0.1:33786)
0619 18:53:42.623882 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:42.623905 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:42.624038 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:42.624350 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:42.624354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:42.624735 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:42.624891 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.624900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.626027 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.626032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.627103 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.627119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.627901 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.627906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.628039 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.628636 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:42.628654 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:42.630946 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:42.633926 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.633932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.633936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.634403 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.634933 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.634938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.634942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.635098 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.635669 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:42.635674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:42.635834 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 18:53:42.635985 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 18:53:42.636379 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260619 18:53:42.636742 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.623525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.15.91.61:35667)
0619 18:53:42.623730 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.623735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.624013 (+ 278us) server_negotiation.cc:407] Connection header received
0619 18:53:42.624149 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.624153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.624225 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.624310 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:42.625121 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.625882 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.627287 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.627726 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.627842 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.631203 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:42.631209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:42.631213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:42.631251 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:42.633717 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.634552 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.634558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.634561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.634785 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.635221 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.635226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.635230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.635536 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:42.635706 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.636093 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.636378 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:53:42.637993 22619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:53:42 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895222, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.60@KRBTEST.COM for oryx/127.15.91.60@KRBTEST.COM
I20260619 18:53:42.651623 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.638783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:43887 (local address 127.0.0.1:55152)
0619 18:53:42.639024 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:42.639041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:42.639143 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:42.639431 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:42.639434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:42.639804 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:42.639974 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.639982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.640985 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.640989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.641868 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.641882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.642587 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.642591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.642837 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.643528 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:42.643547 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:42.645701 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:42.648953 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.648958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.648960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.649380 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.649860 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.649865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.649868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.650047 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.650609 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:42.650613 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:42.650845 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 18:53:42.651017 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 18:53:42.651268 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 18:53:42.651715 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.638829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.15.91.60:43887)
0619 18:53:42.638995 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.639000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.639106 (+ 106us) server_negotiation.cc:407] Connection header received
0619 18:53:42.639238 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.639241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.639300 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.639387 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:42.640151 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.640858 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.642021 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.642422 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.642539 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.645931 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:42.645937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:42.645941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:42.645975 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:42.648752 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.649579 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.649583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.649585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.649710 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.650157 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.650160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.650162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.650476 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:42.650607 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.651088 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.651410 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":52}
I20260619 18:53:42.657399 22619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Bootstrap starting.
I20260619 18:53:42.660984 22619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:42.662117 22619 log.cc:824] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:42.665215 22619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: No bootstrap required, opened a new log
I20260619 18:53:42.672996 22619 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } }
I20260619 18:53:42.673395 22619 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:42.673544 22619 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccea9e7e30e34087952516eb8a4ac28b, State: Initialized, Role: FOLLOWER
I20260619 18:53:42.674227 22619 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } }
I20260619 18:53:42.675392 22626 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } } }
I20260619 18:53:42.675735 22626 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:53:42.676571 22619 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:53:42.687922 22637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:53:42.688133 22637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:53:42.694207 22462 master_runner.cc:428] Detected that this master ccea9e7e30e34087952516eb8a4ac28b is joining an existing cluster
I20260619 18:53:42.694377 22462 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.91.60:43887
I20260619 18:53:42.719483 22626 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:42.719874 22626 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } }
I20260619 18:53:42.721904 22626 leader_election.cc:288] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68bcf1108b5f49d398f22fe37bc8c056 (127.15.91.62:34269), 3943f18ae29c4d5a99391d6b77d60a8e (127.15.91.61:35667)
I20260619 18:53:42.722679 21910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccea9e7e30e34087952516eb8a4ac28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68bcf1108b5f49d398f22fe37bc8c056" is_pre_election: true
I20260619 18:53:42.722779 22511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccea9e7e30e34087952516eb8a4ac28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" is_pre_election: true
I20260619 18:53:42.723855 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.699752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.15.91.61:35667)
0619 18:53:42.699975 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.699982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.700782 (+ 800us) server_negotiation.cc:407] Connection header received
0619 18:53:42.701113 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.701120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.701213 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.701294 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:42.703108 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.704251 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.706783 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.707479 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.707606 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.710868 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:42.710876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:42.710882 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:42.710943 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:42.715170 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.716242 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.716250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.716254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.716485 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.717085 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.717091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.717095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.717506 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:42.717647 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.718472 (+ 825us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.718827 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 18:53:42.723827 22568 leader_election.cc:302] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccea9e7e30e34087952516eb8a4ac28b; no voters: 3943f18ae29c4d5a99391d6b77d60a8e, 68bcf1108b5f49d398f22fe37bc8c056
I20260619 18:53:42.725090 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.699658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35667 (local address 127.0.0.1:33792)
0619 18:53:42.700648 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:42.700675 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:42.700933 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:42.701557 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:42.701562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:42.702373 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:42.702833 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.702850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.704515 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.704521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.706517 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.706537 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.707975 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.707984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.708189 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.709084 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:42.709110 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:42.710560 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:42.715401 (+ 4841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.715408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.715413 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.716044 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.716688 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.716694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.716698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.716928 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.717765 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:42.717771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:42.718106 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 18:53:42.724156 (+ 6050us) client_negotiation.cc:240] Negotiation successful
0619 18:53:42.724581 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":540,"threads_started":1}
I20260619 18:53:42.726197 22626 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 18:53:42.726497 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 21961
I20260619 18:53:42.733255 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.699622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.15.91.62:34269)
0619 18:53:42.699834 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.699841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.705350 (+ 5509us) server_negotiation.cc:407] Connection header received
0619 18:53:42.705586 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.705597 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.705717 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.705841 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:42.708081 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.710159 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.714696 (+ 4537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.715759 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.716991 (+ 1232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.721109 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:42.721119 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:42.721125 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:42.721175 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:42.728498 (+ 7323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.729583 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.729593 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.729596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.729795 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.731201 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.731206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.731209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.731653 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:42.731857 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.732581 (+ 724us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.732870 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:53:42.734655 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.700501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:44600)
0619 18:53:42.705191 (+ 4690us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:42.705210 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:42.705420 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:42.706110 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:42.706114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:42.707203 (+ 1089us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:42.707769 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.707790 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.710683 (+ 2893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.710756 (+ 73us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.714318 (+ 3562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.714368 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.717748 (+ 3380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.717757 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.718116 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.719066 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:42.719094 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:42.720780 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:42.728744 (+ 7964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.728752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.728759 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.729383 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.729966 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.729972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.729977 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.731022 (+ 1045us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.731960 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:42.731965 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:42.732236 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 18:53:42.733561 (+ 1325us) client_negotiation.cc:240] Negotiation successful
0619 18:53:42.734092 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4556,"thread_start_us":19483,"threads_started":1}
Jun 19 18:53:42 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.91.1@KRBTEST.COM
I20260619 18:53:42.750514 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.700887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:43887 (local address 127.0.0.1:55164)
0619 18:53:42.719084 (+ 18197us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:42.719106 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:42.719296 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:42.719833 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:42.719837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:42.720780 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:42.721256 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.721270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.723296 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.723303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.728689 (+ 5386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.728709 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.730095 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.730103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.730303 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.732422 (+ 2119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:42.732451 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:42.734156 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:42.744593 (+ 10437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.744602 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.744609 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.745460 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.747273 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.747282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.747288 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.747537 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.748656 (+ 1119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:42.748664 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:42.748926 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 18:53:42.749129 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 18:53:42.749647 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18015,"thread_start_us":3892,"threads_started":1}
I20260619 18:53:42.752151 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.701271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.15.91.60:43887)
0619 18:53:42.701545 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.701567 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.719192 (+ 17625us) server_negotiation.cc:407] Connection header received
0619 18:53:42.719444 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.719453 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.719535 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.719613 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:42.721430 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.722648 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.728852 (+ 6204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.729637 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.729742 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.734389 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:42.734396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:42.734402 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:42.734442 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:42.744354 (+ 9912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.745698 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.745708 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.745714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.746541 (+ 827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.747716 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.747725 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.747731 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.748301 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:42.748539 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.751196 (+ 2657us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.751586 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:53:42.756238 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.728524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.15.91.1:35739)
0619 18:53:42.729274 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.729284 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.729317 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:53:42.729388 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.729396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.729715 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.731832 (+ 2117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:42.732135 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.733789 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.736070 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.737165 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.738601 (+ 1436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.744666 (+ 6065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:42.744695 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:42.744720 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:42.744784 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:42.749562 (+ 4778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.751553 (+ 1991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.751565 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.751569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.751820 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.752794 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.752799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.752803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.753382 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:42.755158 (+ 1776us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.755380 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.755755 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":403,"threads_started":1}
I20260619 18:53:42.758342 22084 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 19 18:53:42 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895219, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.62@KRBTEST.COM for oryx/127.15.91.60@KRBTEST.COM
I20260619 18:53:42.777204 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.742893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:43887 (local address 127.0.0.1:55174)
0619 18:53:42.743238 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:42.743265 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:42.743436 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:42.755497 (+ 12061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:42.755506 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:42.756278 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:42.756644 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.756666 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.759422 (+ 2756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.759434 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.761712 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.761744 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.764141 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.764152 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.764463 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.765513 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:42.765570 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:42.768350 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:42.773570 (+ 5220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.773580 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.773585 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.774265 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.774890 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.774896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.774910 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.775098 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.775967 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:42.775975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:42.776169 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 18:53:42.776368 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 18:53:42.776740 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260619 18:53:42.777410 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.742910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.15.91.60:43887)
0619 18:53:42.754826 (+ 11916us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.754833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.754854 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:53:42.754988 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.754997 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.755140 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.756905 (+ 1765us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:42.757216 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.758950 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.762597 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.763814 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.766920 (+ 3106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.768643 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:42.768650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:42.768653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:42.768692 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:42.773307 (+ 4615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.774504 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.774512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.774516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.774693 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.775273 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.775279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.775283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.775789 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:42.776542 (+ 753us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.776702 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.776989 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11735,"thread_start_us":23084,"threads_started":1}
I20260619 18:53:42.779331 21900 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.91.60:43887: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b"
member_type: NON_VOTER
last_known_addr {
host: "127.15.91.60"
port: 43887
}
attrs {
promote: true
}
}
dest_uuid: "68bcf1108b5f49d398f22fe37bc8c056"
cas_config_opid_index: 5
I20260619 18:53:42.780660 21900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } }
I20260619 18:53:42.784008 22460 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ccea9e7e30e34087952516eb8a4ac28b (127.15.91.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:42.791800 22594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:42.796492 22511 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 LEARNER]: Refusing update from remote peer 68bcf1108b5f49d398f22fe37bc8c056: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 18:53:42.796883 22594 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 FOLLOWER]: Refusing update from remote peer 68bcf1108b5f49d398f22fe37bc8c056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 18:53:42.797664 22651 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 18:53:42.798024 22460 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.91.60:43887
I20260619 18:53:42.798779 22462 master.cc:537] Master@127.15.91.60:43887 shutting down...
I20260619 18:53:42.803974 22511 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ccea9e7e30e34087952516eb8a4ac28b (127.15.91.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:42.809237 22539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:42.809628 22539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: This master's current role is: LEARNER
I20260619 18:53:42.809796 22649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: NON_VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: true } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:42.810256 22649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:42.815286 22460 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056: attempting to promote NON_VOTER 3943f18ae29c4d5a99391d6b77d60a8e to VOTER
I20260619 18:53:42.815615 22650 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:42.820595 22460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } }
I20260619 18:53:42.823256 22511 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 LEARNER]: Refusing update from remote peer 68bcf1108b5f49d398f22fe37bc8c056: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 18:53:42.824055 22649 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 18:53:42.827461 22650 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3943f18ae29c4d5a99391d6b77d60a8e (127.15.91.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:42.831131 22511 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3943f18ae29c4d5a99391d6b77d60a8e (127.15.91.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:42.830528 22650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:42.834569 22650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:42.834245 22460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:42.834874 22460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:42.835269 22539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:42.835542 22539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: This master's current role is: FOLLOWER
W20260619 18:53:42.839169 22573 tablet.cc:2369] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:53:42.843206 22462 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:42.843719 22462 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:42.843955 22462 tablet_replica.cc:330] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: stopping tablet replica
W20260619 18:53:42.845408 21884 consensus_peers.cc:487] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 -> Peer ccea9e7e30e34087952516eb8a4ac28b (127.15.91.60:43887): Couldn't send request to peer ccea9e7e30e34087952516eb8a4ac28b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 18:53:42.852285 22462 master.cc:552] Master@127.15.91.60:43887 shutdown complete.
I20260619 18:53:42.852412 22462 master_runner.cc:305] Clearing existing system tablet
I20260619 18:53:42.852874 22462 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:42.853039 22462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:42.862594 22462 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.006s sys 0.005s
I20260619 18:53:42.867228 22656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:42.867877 22462 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:42.868227 22462 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260619 18:53:42.868361 22462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/wal
uuid: "ccea9e7e30e34087952516eb8a4ac28b"
format_stamp: "Formatted at 2026-06-19 18:53:42 on dist-test-slave-dm3f"
server_key: "ccb39b53205645b47f6672d1375e4cfb"
server_key_iv: "18f00770f4b80c55d9621590c5328831"
server_key_version: "encryptionkey@0"
I20260619 18:53:42.874264 22462 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:42.882586 22462 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 18:53:42.882819 22462 log.cc:1196] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 18:53:42.883385 22462 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Deleting consensus metadata
I20260619 18:53:42.883678 22462 master_runner.cc:315] Copying system tablet from 127.15.91.62:34269
I20260619 18:53:42.894930 22462 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: tablet copy: Beginning tablet copy session from remote peer at address 127.15.91.62:34269
I20260619 18:53:42.920332 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.895917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.15.91.62:34269)
0619 18:53:42.896206 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:42.896213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:42.903530 (+ 7317us) server_negotiation.cc:407] Connection header received
0619 18:53:42.903734 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:42.903741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:42.903843 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:42.903969 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:42.905190 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.906385 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.908492 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.909188 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.909341 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.912090 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:42.912098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:42.912102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:42.912146 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:42.915927 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.916975 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.916981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.916984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.917271 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:42.918036 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:42.918044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:42.918051 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:42.918676 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:42.918913 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:42.919636 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 18:53:42.919930 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260619 18:53:42.920342 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:42.895891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:44616)
0619 18:53:42.903318 (+ 7427us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:42.903362 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:42.903567 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:42.904031 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:42.904036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:42.904668 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:42.904961 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.904980 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.906583 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.906589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.908267 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:42.908288 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:42.909422 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:42.909429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:42.909641 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:42.910458 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:42.910485 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:42.911771 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:42.916161 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.916170 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.916177 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.916770 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.917520 (+ 750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:42.917528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:42.917536 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:42.917765 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:42.919006 (+ 1241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:42.919015 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:42.919274 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 18:53:42.919444 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 18:53:42.919834 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7231,"thread_start_us":7046,"threads_started":1}
I20260619 18:53:42.920753 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 21961
I20260619 18:53:42.922319 21920 tablet_copy_service.cc:140] P 68bcf1108b5f49d398f22fe37bc8c056: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ccea9e7e30e34087952516eb8a4ac28b ({username='oryx', principal='oryx/127.15.91.60@KRBTEST.COM'} at 127.0.0.1:44616)
I20260619 18:53:42.922555 21920 tablet_copy_service.cc:161] P 68bcf1108b5f49d398f22fe37bc8c056: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ccea9e7e30e34087952516eb8a4ac28b at {username='oryx', principal='oryx/127.15.91.60@KRBTEST.COM'} at 127.0.0.1:44616: session id = ccea9e7e30e34087952516eb8a4ac28b-00000000000000000000000000000000
I20260619 18:53:42.928664 21920 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 18:53:42.930987 22462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 18:53:42.938587 22462 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: tablet copy: Starting download of 0 data blocks...
I20260619 18:53:42.939002 22462 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: tablet copy: Starting download of 1 WAL segments...
I20260619 18:53:42.941892 22462 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 18:53:42.942840 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:35739
--local_ip_for_outbound_sockets=127.15.91.1
--tserver_master_addrs=127.15.91.62:34269
--webserver_port=33191
--webserver_interface=127.15.91.1
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab
--principal=oryx/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.91.62:34269,127.15.91.61:35667
--tserver_master_addrs=127.15.91.62:34269,127.15.91.61:35667,127.15.91.60:43887 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:53:42.947140 21920 tablet_copy_service.cc:342] P 68bcf1108b5f49d398f22fe37bc8c056: Request end of tablet copy session ccea9e7e30e34087952516eb8a4ac28b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.91.60@KRBTEST.COM'} at 127.0.0.1:44616
I20260619 18:53:42.947343 21920 tablet_copy_service.cc:433] P 68bcf1108b5f49d398f22fe37bc8c056: ending tablet copy session ccea9e7e30e34087952516eb8a4ac28b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ccea9e7e30e34087952516eb8a4ac28b
I20260619 18:53:42.961889 22462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:42.965212 22664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:42.966226 22665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:42.974292 22667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:42.974622 22462 server_base.cc:1034] running on GCE node
I20260619 18:53:42.974979 22462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:42.975695 22462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:42.976920 22462 hybrid_clock.cc:648] HybridClock initialized: now 1781895222976848 us; error 68 us; skew 500 ppm
I20260619 18:53:42.977185 22462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:42 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895222, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:42.979517 22462 init.cc:377] Logged in from keytab as oryx/127.15.91.60@KRBTEST.COM (short username oryx)
I20260619 18:53:42.987193 22462 webserver.cc:460] Webserver started at http://127.15.91.60:43519/ using document root <none> and password file <none>
I20260619 18:53:42.987567 22462 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:42.987684 22462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:42.990293 22462 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:53:42.992273 22674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:42.992736 22462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 18:53:42.992905 22462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/wal
uuid: "ccea9e7e30e34087952516eb8a4ac28b"
format_stamp: "Formatted at 2026-06-19 18:53:42 on dist-test-slave-dm3f"
server_key: "ccb39b53205645b47f6672d1375e4cfb"
server_key_iv: "18f00770f4b80c55d9621590c5328831"
server_key_version: "encryptionkey@0"
I20260619 18:53:42.993095 22462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:43.016654 22462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:43.020850 22462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:43.051378 22462 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.60:43887
I20260619 18:53:43.051378 22725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.60:43887 every 8 connection(s)
I20260619 18:53:43.052393 22462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
I20260619 18:53:43.060061 22726 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:53:43.064677 22726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Bootstrap starting.
I20260619 18:53:43.082276 22726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:53:43.083072 22726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Bootstrap complete.
I20260619 18:53:43.084437 22726 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } }
I20260619 18:53:43.084884 22726 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ccea9e7e30e34087952516eb8a4ac28b, State: Initialized, Role: LEARNER
I20260619 18:53:43.085201 22726 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } }
I20260619 18:53:43.086246 22729 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:43.086645 22726 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:43.086741 22729 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: This master's current role is: LEARNER
I20260619 18:53:43.108188 22740 catalog_manager.cc:1259] Loaded cluster ID: 40b587c88d5540ceb49e34cfd5421bbf
I20260619 18:53:43.108345 22740 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: loading cluster ID for follower catalog manager: success
I20260619 18:53:43.111083 22740 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: acquiring CA information for follower catalog manager: success
I20260619 18:53:43.113169 22740 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:53:43.413321 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:43.406104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.15.91.60:43887)
0619 18:53:43.406771 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:43.406784 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:43.406804 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:53:43.406869 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:43.406873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:43.406949 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:43.407057 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:43.407603 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.408934 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.411406 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.412241 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.412391 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:43.412562 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:43.412922 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 18:53:43.413039 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":357,"threads_started":1}
I20260619 18:53:43.413409 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:43.405852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:43887 (local address 127.0.0.1:55176)
0619 18:53:43.406401 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:43.406428 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:43.406606 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:43.407131 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:43.407138 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:43.407169 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:43.407398 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:43.407412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.409127 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.409135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:43.411174 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:43.411200 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.412502 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.412509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:43.412716 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:43.412756 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:53:43.412881 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 18:53:43.413049 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":264,"threads_started":1}
I20260619 18:53:43.414381 22701 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 18:53:43.415146 22729 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68bcf1108b5f49d398f22fe37bc8c056. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:43.415481 22729 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: This master's current role is: LEARNER
W20260619 18:53:43.418169 22663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:43.418532 22663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:43.418804 22663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:43.427035 22663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:43.427212 22663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:43.427409 22663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:53:43.438149 22663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab
--principal=oryx/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:35739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=33191
--tserver_master_addrs=127.15.91.62:34269,127.15.91.61:35667,127.15.91.60:43887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:43.439749 22663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:43.441264 22663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:43.456429 22748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:43.456509 22750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:43.456512 22747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:43.457628 22663 server_base.cc:1034] running on GCE node
I20260619 18:53:43.458279 22663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:43.459399 22663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:43.460633 22663 hybrid_clock.cc:648] HybridClock initialized: now 1781895223460588 us; error 41 us; skew 500 ppm
I20260619 18:53:43.461027 22663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:43 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895223, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:43.464531 22663 init.cc:377] Logged in from keytab as oryx/127.15.91.1@KRBTEST.COM (short username oryx)
I20260619 18:53:43.466292 22663 webserver.cc:460] Webserver started at http://127.15.91.1:33191/ using document root <none> and password file <none>
I20260619 18:53:43.467257 22663 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:43.467444 22663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:43.474126 22663 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260619 18:53:43.477907 22757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:43.479826 22663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260619 18:53:43.480082 22663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "d1f612cc551a4147868957b40b15baf8"
format_stamp: "Formatted at 2026-06-19 18:53:40 on dist-test-slave-dm3f"
server_key: "dd356c2658e9d91ee9ad38463e45fa14"
server_key_iv: "376b1465a356a1567aec9af231337938"
server_key_version: "encryptionkey@0"
I20260619 18:53:43.481395 22663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:43.502985 22663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:43.508409 22663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:43.508701 22663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:43.510164 22663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:43.511976 22663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:43.512094 22663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:43.512183 22663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:43.512250 22663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:43.554229 22663 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:35739
I20260619 18:53:43.554348 22869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:35739 every 8 connection(s)
I20260619 18:53:43.556035 22663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:53:43.559731 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 22663
I20260619 18:53:43.560181 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 22095
Jun 19 18:53:43 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895223, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
Jun 19 18:53:43 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895223, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.61@KRBTEST.COM
Jun 19 18:53:43 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895223, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.60@KRBTEST.COM
Jun 19 18:53:43 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.91.2@KRBTEST.COM
I20260619 18:53:43.598755 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:43.563299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:45509 (local address 127.15.91.60:43887)
0619 18:53:43.563515 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:43.563522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:43.572713 (+ 9191us) server_negotiation.cc:407] Connection header received
0619 18:53:43.572900 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:43.572909 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:43.573010 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:43.573118 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:43.576248 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.577497 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.580804 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.581543 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.581661 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:43.589923 (+ 8262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:43.589929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:43.589933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:43.590009 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:43.594422 (+ 4413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:43.595569 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:43.595578 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:43.595580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:43.595751 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:43.596262 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:43.596267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:43.596270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:43.596668 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:43.596815 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:43.598045 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0619 18:53:43.598349 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:53:43.599259 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:43.561129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:32981 (local address 127.15.91.61:35667)
0619 18:53:43.574917 (+ 13788us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:43.574923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:43.574953 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:53:43.575017 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:43.575024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:43.575125 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:43.576832 (+ 1707us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:43.577041 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.578279 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.580815 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.581526 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.584159 (+ 2633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:43.589040 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:43.589049 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:43.589057 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:43.589111 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:43.594005 (+ 4894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:43.595231 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:43.595237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:43.595240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:43.595435 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:43.596103 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:43.596110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:43.596114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:43.596571 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:43.597011 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:43.598567 (+ 1556us) server_negotiation.cc:299] Negotiation successful
0619 18:53:43.598846 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13659,"thread_start_us":13513,"threads_started":1}
I20260619 18:53:43.600478 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:43.560492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35667 (local address 127.15.91.1:32981)
0619 18:53:43.561637 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:43.561712 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:43.566966 (+ 5254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:43.575788 (+ 8822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:43.575796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:43.576365 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:43.576650 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:43.576663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.578491 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.578496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:43.580608 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:43.580624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.581834 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.581845 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:43.582215 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:43.583417 (+ 1202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:43.583434 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:43.588694 (+ 5260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:43.594185 (+ 5491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:43.594191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:43.594194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:43.595049 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:43.595595 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:43.595601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:43.595607 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:43.595922 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:43.596798 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:43.596810 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:43.597472 (+ 662us) client_negotiation.cc:769] Sending connection context
0619 18:53:43.598251 (+ 779us) client_negotiation.cc:240] Negotiation successful
0619 18:53:43.598643 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":381,"spinlock_wait_cycles":59520,"thread_start_us":685,"threads_started":1}
I20260619 18:53:43.600524 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:43.563541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:43887 (local address 127.15.91.1:45509)
0619 18:53:43.572595 (+ 9054us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:43.572610 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:43.572758 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:43.573701 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:43.573716 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:43.574875 (+ 1159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:43.575969 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:43.576009 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.577850 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.577857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:43.580608 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:43.580624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.582053 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.582060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:43.582213 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:43.583359 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:43.583400 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:43.589660 (+ 6260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:43.594655 (+ 4995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:43.594664 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:43.594670 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:43.595398 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:43.595979 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:43.595983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:43.595988 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:43.596131 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:43.596894 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:43.596900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:43.597678 (+ 778us) client_negotiation.cc:769] Sending connection context
0619 18:53:43.597851 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 18:53:43.599828 (+ 1977us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8912,"mutex_wait_us":113,"thread_start_us":8009,"threads_started":1}
I20260619 18:53:43.601068 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:43.561750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.15.91.2:39293)
0619 18:53:43.571415 (+ 9665us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:43.571423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:43.573671 (+ 2248us) server_negotiation.cc:407] Connection header received
0619 18:53:43.573763 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:43.573772 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:43.574079 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:43.574275 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:43.577969 (+ 3694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.579898 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.582176 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.583197 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.583333 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:43.591314 (+ 7981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:43.591341 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:43.591363 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:43.591418 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:43.596236 (+ 4818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:43.597424 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:43.597435 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:43.597439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:43.597660 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:43.598164 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:43.598170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:43.598173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:43.598637 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:43.599063 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:43.600316 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0619 18:53:43.600555 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9466,"thread_start_us":9273,"threads_started":1}
I20260619 18:53:43.602259 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:43.559752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:59485 (local address 127.15.91.62:34269)
0619 18:53:43.561022 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:43.561029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:43.575274 (+ 14245us) server_negotiation.cc:407] Connection header received
0619 18:53:43.575367 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:43.575372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:43.575452 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:43.575564 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:43.578471 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.579605 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.582017 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.582765 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.582932 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:43.589074 (+ 6142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:43.589082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:43.589086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:43.589132 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:43.593577 (+ 4445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:43.595459 (+ 1882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:43.595466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:43.595470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:43.595659 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:43.596229 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:43.596235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:43.596238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:43.596668 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:43.596815 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:43.601545 (+ 4730us) server_negotiation.cc:299] Negotiation successful
0619 18:53:43.601855 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1110,"thread_start_us":969,"threads_started":1}
I20260619 18:53:43.602617 22218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:43.602818 22872 heartbeater.cc:347] Connected to a master server at 127.15.91.61:35667
I20260619 18:53:43.602869 22870 heartbeater.cc:347] Connected to a master server at 127.15.91.60:43887
I20260619 18:53:43.603008 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:43.560492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.1:59485)
0619 18:53:43.571916 (+ 11424us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:43.571941 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:43.572130 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:43.577189 (+ 5059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:43.577196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:43.577795 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:43.578235 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:43.578249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.579778 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.579791 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:43.581693 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:43.581718 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:43.583249 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:43.583261 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:43.583515 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:43.584345 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:43.584376 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:43.588680 (+ 4304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:43.593883 (+ 5203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:43.593900 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:43.593926 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:43.595249 (+ 1323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:43.595843 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:43.595852 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:43.595856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:43.596083 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:43.596894 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:43.596900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:43.597472 (+ 572us) client_negotiation.cc:769] Sending connection context
0619 18:53:43.597851 (+ 379us) client_negotiation.cc:240] Negotiation successful
0619 18:53:43.602467 (+ 4616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11242,"mutex_wait_us":1212,"spinlock_wait_cycles":1408,"thread_start_us":13147,"threads_started":1}
I20260619 18:53:43.603214 22872 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:43.603210 22870 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:43.604072 22871 heartbeater.cc:347] Connected to a master server at 127.15.91.62:34269
I20260619 18:53:43.604233 22871 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:43.604586 22870 heartbeater.cc:511] Master 127.15.91.60:43887 requested a full tablet report, sending...
I20260619 18:53:43.604753 22871 heartbeater.cc:511] Master 127.15.91.62:34269 requested a full tablet report, sending...
I20260619 18:53:43.605334 22872 heartbeater.cc:511] Master 127.15.91.61:35667 requested a full tablet report, sending...
I20260619 18:53:43.606972 21900 ts_manager.cc:194] Re-registered known tserver with Master: d1f612cc551a4147868957b40b15baf8 (127.15.91.1:35739)
I20260619 18:53:43.607623 22501 ts_manager.cc:194] Registered new tserver with Master: d1f612cc551a4147868957b40b15baf8 (127.15.91.1:35739)
I20260619 18:53:43.608178 21900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:59485
I20260619 18:53:43.609786 22691 ts_manager.cc:194] Registered new tserver with Master: d1f612cc551a4147868957b40b15baf8 (127.15.91.1:35739)
I20260619 18:53:43.694823 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 22095
I20260619 18:53:43.713006 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:39293
--local_ip_for_outbound_sockets=127.15.91.2
--tserver_master_addrs=127.15.91.62:34269
--webserver_port=41867
--webserver_interface=127.15.91.2
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab
--principal=oryx/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.91.62:34269,127.15.91.61:35667
--tserver_master_addrs=127.15.91.62:34269,127.15.91.61:35667,127.15.91.60:43887 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:53:43.886348 22650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: NON_VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: true } } }
I20260619 18:53:43.886698 22650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:43.888232 22884 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056: attempting to promote NON_VOTER ccea9e7e30e34087952516eb8a4ac28b to VOTER
I20260619 18:53:43.888972 22884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } }
I20260619 18:53:43.891652 22701 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 LEARNER]: Refusing update from remote peer 68bcf1108b5f49d398f22fe37bc8c056: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 18:53:43.891741 22511 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 FOLLOWER]: Refusing update from remote peer 68bcf1108b5f49d398f22fe37bc8c056: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 18:53:43.892594 22650 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 18:53:43.893049 22884 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 18:53:43.896684 22883 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ccea9e7e30e34087952516eb8a4ac28b (127.15.91.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:43.900148 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:43.900477 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:43.900863 22511 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ccea9e7e30e34087952516eb8a4ac28b (127.15.91.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:43.903038 22650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:43.903539 22650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:43.907320 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:43.907896 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:53:43.910071 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:43.910446 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:43.911285 22701 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ccea9e7e30e34087952516eb8a4ac28b (127.15.91.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:43.917240 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68bcf1108b5f49d398f22fe37bc8c056" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:43.917600 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: This master's current role is: FOLLOWER
W20260619 18:53:44.108083 22882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:44.108464 22882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:44.108620 22882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:44.116729 22882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:44.116927 22882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:44.117102 22882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:53:44.127820 22882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab
--principal=oryx/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:39293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=41867
--tserver_master_addrs=127.15.91.62:34269,127.15.91.61:35667,127.15.91.60:43887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:44.129331 22882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:44.130818 22882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:44.145685 22895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:44.145681 22896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:44.145768 22898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:44.146625 22882 server_base.cc:1034] running on GCE node
I20260619 18:53:44.147363 22882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:44.148476 22882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:44.149742 22882 hybrid_clock.cc:648] HybridClock initialized: now 1781895224149691 us; error 49 us; skew 500 ppm
I20260619 18:53:44.150130 22882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:44 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895224, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:44.153789 22882 init.cc:377] Logged in from keytab as oryx/127.15.91.2@KRBTEST.COM (short username oryx)
I20260619 18:53:44.155766 22882 webserver.cc:460] Webserver started at http://127.15.91.2:41867/ using document root <none> and password file <none>
I20260619 18:53:44.156658 22882 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:44.156853 22882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:44.163770 22882 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:53:44.167224 22905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:44.168987 22882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260619 18:53:44.169250 22882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "ebb4667c5c934fd090783186dd64a3a2"
format_stamp: "Formatted at 2026-06-19 18:53:40 on dist-test-slave-dm3f"
server_key: "6c17d41701e635c127468468b82ffd73"
server_key_iv: "79910c692e01b44a5aa4508f9aacdfdf"
server_key_version: "encryptionkey@0"
I20260619 18:53:44.170409 22882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:44.199012 22882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:44.204124 22882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:44.204489 22882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:44.205672 22882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:44.207388 22882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:44.207551 22882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:44.207691 22882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:44.207787 22882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:44.246268 22882 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:39293
I20260619 18:53:44.246330 23017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:39293 every 8 connection(s)
I20260619 18:53:44.247891 22882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:53:44.250023 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 22882
I20260619 18:53:44.250317 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 22229
Jun 19 18:53:44 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895224, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
Jun 19 18:53:44 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895224, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.60@KRBTEST.COM
Jun 19 18:53:44 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895224, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.61@KRBTEST.COM
Jun 19 18:53:44 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.91.3@KRBTEST.COM
I20260619 18:53:44.295744 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.252531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:33699 (local address 127.15.91.62:34269)
0619 18:53:44.269140 (+ 16609us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:44.269151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:44.269183 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:53:44.269277 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:44.269286 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:44.269399 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:44.269574 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:44.271875 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.273616 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.277244 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.277923 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.278086 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:44.284266 (+ 6180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:44.284272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:44.284275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:44.284313 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:44.288414 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:44.289899 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:44.289907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:44.289913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:44.290118 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:44.290999 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:44.291006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:44.291011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:44.291454 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:44.291635 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:44.294986 (+ 3351us) server_negotiation.cc:299] Negotiation successful
0619 18:53:44.295290 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16387,"thread_start_us":18211,"threads_started":1}
I20260619 18:53:44.296669 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.252191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.2:33699)
0619 18:53:44.266993 (+ 14802us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:44.267033 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:44.267199 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:44.269853 (+ 2654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:44.269862 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:44.271033 (+ 1171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:44.271562 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:44.271585 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.273963 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.273973 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:44.277049 (+ 3076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:44.277068 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.278449 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.278457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:44.278904 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:44.279917 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:44.279960 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:44.283939 (+ 3979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:44.288668 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:44.288682 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:44.288709 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:44.289673 (+ 964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:44.290430 (+ 757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:44.290437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:44.290442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:44.290829 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:44.291727 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:44.291740 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:44.292286 (+ 546us) client_negotiation.cc:769] Sending connection context
0619 18:53:44.292670 (+ 384us) client_negotiation.cc:240] Negotiation successful
0619 18:53:44.293319 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8565,"mutex_wait_us":4,"spinlock_wait_cycles":23424,"thread_start_us":8227,"threads_started":1}
I20260619 18:53:44.296772 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.262978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:46037 (local address 127.15.91.61:35667)
0619 18:53:44.263817 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:44.263823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:44.263847 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:53:44.266310 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:44.266316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:44.266421 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:44.266582 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:44.278340 (+ 11758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.279789 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.282361 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.283175 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.283287 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:44.288392 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:44.288400 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:44.288404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:44.288469 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:44.293119 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:44.294181 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:44.294190 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:44.294196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:44.294395 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:44.294943 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:44.294947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:44.294950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:44.295323 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:44.295482 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:44.296137 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 18:53:44.296432 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":569,"threads_started":1}
I20260619 18:53:44.297664 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.262457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35667 (local address 127.15.91.2:46037)
0619 18:53:44.263235 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:44.263263 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:44.265786 (+ 2523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:44.267411 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:44.267431 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:44.268825 (+ 1394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:44.270177 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:44.270228 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.280156 (+ 9928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.280164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:44.282141 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:44.282159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.283684 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.283691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:44.283891 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:44.284696 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:44.284720 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:44.287966 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:44.293414 (+ 5448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:44.293419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:44.293435 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:44.294029 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:44.294627 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:44.294634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:44.294639 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:44.294803 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:44.295558 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:44.295577 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:44.295715 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 18:53:44.295834 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 18:53:44.297196 (+ 1362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":983,"thread_start_us":1200,"threads_started":1}
I20260619 18:53:44.297736 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.251496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:33907 (local address 127.15.91.60:43887)
0619 18:53:44.275498 (+ 24002us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:44.275505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:44.275531 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:53:44.275607 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:44.275615 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:44.275714 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:44.275866 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:44.277345 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.278646 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.280997 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.281713 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.281848 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:44.288474 (+ 6626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:44.288482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:44.288486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:44.288538 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:44.292697 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:44.293754 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:44.293761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:44.293767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:44.293953 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:44.294576 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:44.294583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:44.294586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:44.295062 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:44.295205 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:44.296739 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0619 18:53:44.297330 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23851,"thread_start_us":23676,"threads_started":1}
I20260619 18:53:44.297061 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.252191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:43887 (local address 127.15.91.2:33907)
0619 18:53:44.261818 (+ 9627us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:44.261906 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:44.265788 (+ 3882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:44.276124 (+ 10336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:44.276130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:44.276768 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:44.277114 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:44.277131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.278907 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.278913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:44.280783 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:44.280799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.282171 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.282179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:44.282385 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:44.283205 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:44.283232 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:44.288215 (+ 4983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:44.292905 (+ 4690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:44.292911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:44.292918 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:44.293588 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:44.294099 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:44.294105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:44.294109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:44.294370 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:44.295282 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:44.295287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:44.295447 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 18:53:44.295592 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 18:53:44.296137 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9244,"mutex_wait_us":828,"thread_start_us":8761,"threads_started":1}
I20260619 18:53:44.299268 23020 heartbeater.cc:347] Connected to a master server at 127.15.91.61:35667
I20260619 18:53:44.299669 23020 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:44.300220 23018 heartbeater.cc:347] Connected to a master server at 127.15.91.60:43887
I20260619 18:53:44.300361 23018 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:44.300827 23020 heartbeater.cc:511] Master 127.15.91.61:35667 requested a full tablet report, sending...
I20260619 18:53:44.300849 23018 heartbeater.cc:511] Master 127.15.91.60:43887 requested a full tablet report, sending...
I20260619 18:53:44.301666 23019 heartbeater.cc:347] Connected to a master server at 127.15.91.62:34269
I20260619 18:53:44.301806 23019 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:44.302349 23019 heartbeater.cc:511] Master 127.15.91.62:34269 requested a full tablet report, sending...
I20260619 18:53:44.302592 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.251847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.15.91.3:38517)
0619 18:53:44.275585 (+ 23738us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:44.275595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:44.275926 (+ 331us) server_negotiation.cc:407] Connection header received
0619 18:53:44.276113 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:44.276124 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:44.276437 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:44.276643 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:44.278880 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.281169 (+ 2289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.283775 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.285974 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.286150 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:44.293555 (+ 7405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:44.293576 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:44.293593 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:44.293645 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:44.298854 (+ 5209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:44.299910 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:44.299921 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:44.299924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:44.300103 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:44.300669 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:44.300674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:44.300678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:44.301119 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:44.301457 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:44.301805 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 18:53:44.302113 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23555,"thread_start_us":22990,"threads_started":1}
I20260619 18:53:44.303078 22691 ts_manager.cc:194] Registered new tserver with Master: ebb4667c5c934fd090783186dd64a3a2 (127.15.91.2:39293)
I20260619 18:53:44.303622 21900 ts_manager.cc:194] Re-registered known tserver with Master: ebb4667c5c934fd090783186dd64a3a2 (127.15.91.2:39293)
I20260619 18:53:44.304545 22352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:44.304827 21900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:33699
I20260619 18:53:44.306013 22501 ts_manager.cc:194] Registered new tserver with Master: ebb4667c5c934fd090783186dd64a3a2 (127.15.91.2:39293)
I20260619 18:53:44.396972 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 22229
I20260619 18:53:44.414597 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:38517
--local_ip_for_outbound_sockets=127.15.91.3
--tserver_master_addrs=127.15.91.62:34269
--webserver_port=33889
--webserver_interface=127.15.91.3
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab
--principal=oryx/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.91.62:34269,127.15.91.61:35667
--tserver_master_addrs=127.15.91.62:34269,127.15.91.61:35667,127.15.91.60:43887 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:53:44.611923 22871 heartbeater.cc:503] Master 127.15.91.62:34269 was elected leader, sending a full tablet report...
W20260619 18:53:44.785634 23031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:44.785974 23031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:44.786139 23031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:44.794790 23031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:44.794972 23031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:44.795164 23031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:53:44.806478 23031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab
--principal=oryx/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:38517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=33889
--tserver_master_addrs=127.15.91.62:34269,127.15.91.61:35667,127.15.91.60:43887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:44.808076 23031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:44.809605 23031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:44.825508 23037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:44.825516 23040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:44.825556 23038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:44.826933 23031 server_base.cc:1034] running on GCE node
I20260619 18:53:44.827764 23031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:44.829041 23031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:44.830371 23031 hybrid_clock.cc:648] HybridClock initialized: now 1781895224830306 us; error 92 us; skew 500 ppm
I20260619 18:53:44.830911 23031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:44 dist-test-slave-dm3f krb5kdc[21855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895224, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:44.834781 23031 init.cc:377] Logged in from keytab as oryx/127.15.91.3@KRBTEST.COM (short username oryx)
I20260619 18:53:44.836643 23031 webserver.cc:460] Webserver started at http://127.15.91.3:33889/ using document root <none> and password file <none>
I20260619 18:53:44.837440 23031 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:44.837595 23031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:44.844151 23031 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:53:44.848948 23047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:44.850994 23031 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:53:44.851260 23031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "a60a1830734746bfa8ab432167bf9cef"
format_stamp: "Formatted at 2026-06-19 18:53:41 on dist-test-slave-dm3f"
server_key: "ee4b9b7b1d7289fff9760da3d802f512"
server_key_iv: "a030c991d1fba318ba84cf6127696ca6"
server_key_version: "encryptionkey@0"
I20260619 18:53:44.852500 23031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:44.878396 23031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:44.883797 23031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:44.884191 23031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:44.885401 23031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:44.887173 23031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:44.887321 23031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:44.887475 23031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:44.887591 23031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:44.925172 23031 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:38517
I20260619 18:53:44.925287 23159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:38517 every 8 connection(s)
I20260619 18:53:44.926965 23031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:53:44.930796 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 23031
Jun 19 18:53:44 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895224, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:45.003804 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.930449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:45067 (local address 127.15.91.62:34269)
0619 18:53:44.972418 (+ 41969us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:44.972428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:44.972453 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:53:44.972528 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:44.972534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:44.972627 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:44.972767 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:44.976045 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.977603 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.981730 (+ 4127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.982885 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.984897 (+ 2012us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:44.990817 (+ 5920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:44.990830 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:44.990838 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:44.990902 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:44.996046 (+ 5144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:44.997434 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:44.997456 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:44.997462 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:44.997648 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:44.998352 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:44.998358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:44.998361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:44.998804 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:44.998975 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.001058 (+ 2083us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.003418 (+ 2360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41794,"thread_start_us":64880,"threads_started":1}
I20260619 18:53:45.003961 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.931317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.3:45067)
0619 18:53:44.932907 (+ 1590us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:44.932983 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:44.936785 (+ 3802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:44.973296 (+ 36511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:44.973317 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:44.974408 (+ 1091us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:44.975639 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:44.975682 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.977931 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.977942 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:44.981342 (+ 3400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:44.981377 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.983439 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.983467 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:44.984066 (+ 599us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:44.985301 (+ 1235us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:44.985350 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:44.990285 (+ 4935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:44.996291 (+ 6006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:44.996304 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:44.996354 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:44.997200 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:44.997810 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:44.997818 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:44.997821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:44.998173 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:44.998976 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:44.998987 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:44.999397 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 18:53:45.000909 (+ 1512us) client_negotiation.cc:240] Negotiation successful
0619 18:53:45.001635 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"thread_start_us":63218,"threads_started":1}
I20260619 18:53:45.008781 23161 heartbeater.cc:347] Connected to a master server at 127.15.91.62:34269
I20260619 18:53:45.009167 23161 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:45.010227 23161 heartbeater.cc:511] Master 127.15.91.62:34269 requested a full tablet report, sending...
Jun 19 18:53:45 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895224, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.60@KRBTEST.COM
I20260619 18:53:45.013175 21900 ts_manager.cc:194] Re-registered known tserver with Master: a60a1830734746bfa8ab432167bf9cef (127.15.91.3:38517)
I20260619 18:53:45.014556 21900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:45067
I20260619 18:53:45.016438 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.942170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.15.91.61:35667)
0619 18:53:44.971553 (+ 29383us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:44.971561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:44.971586 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:53:44.971670 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:44.971679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:44.971795 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:44.976184 (+ 4389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:44.976451 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.978006 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.986874 (+ 8868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.987795 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.987964 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:44.996815 (+ 8851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:44.996822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:44.996826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:44.996873 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:45.003116 (+ 6243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.006575 (+ 3459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.006582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.006588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.006908 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.007498 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.007504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.007507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.008399 (+ 892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:45.015539 (+ 7140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.015719 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.016003 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29274,"thread_start_us":29184,"threads_started":1}
Jun 19 18:53:45 dist-test-slave-dm3f krb5kdc[21855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895224, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.61@KRBTEST.COM
I20260619 18:53:45.022861 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.941682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.15.91.62:34269)
0619 18:53:44.995629 (+ 53947us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:44.995636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:44.995658 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:53:44.995731 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:44.995736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:44.995859 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:44.995998 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:44.998259 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.999596 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.003272 (+ 3676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.004021 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.004190 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.010105 (+ 5915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:45.010112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:45.010117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:45.010165 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:45.015857 (+ 5692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.019330 (+ 3473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.019337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.019340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.019567 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.020103 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.020108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.020111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.020498 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:45.020756 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.021899 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.022167 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53805,"thread_start_us":30427,"threads_started":1}
I20260619 18:53:45.022872 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.953852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.15.91.60:43887)
0619 18:53:44.978651 (+ 24799us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:44.978661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:44.978692 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:53:44.978800 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:44.978809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:44.978934 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:44.982218 (+ 3284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:44.982476 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.984070 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:44.999224 (+ 15154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:44.999933 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.000064 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.006943 (+ 6879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:45.006951 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:45.006956 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:45.006999 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:45.017736 (+ 10737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.018700 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.018706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.018737 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.018966 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.019465 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.019471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.019475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.020785 (+ 1310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:45.021057 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.021837 (+ 780us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.022101 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":875,"server-negotiator.queue_time_us":24602,"spinlock_wait_cycles":9088,"thread_start_us":16027,"threads_started":1}
I20260619 18:53:45.024336 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.930808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:54375 (local address 127.15.91.60:43887)
0619 18:53:44.969868 (+ 39060us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:44.969878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:44.989746 (+ 19868us) server_negotiation.cc:407] Connection header received
0619 18:53:44.996625 (+ 6879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:44.996635 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:44.996725 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:44.996830 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:45.000287 (+ 3457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.001530 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.004278 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.005010 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.005123 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.012711 (+ 7588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:45.012722 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:45.012729 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:45.012783 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:45.017564 (+ 4781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.018708 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.018740 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.018743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.018966 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.019465 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.019471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.019475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.019915 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:45.020064 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.023526 (+ 3462us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.023810 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":38860,"spinlock_wait_cycles":9216,"thread_start_us":47512,"threads_started":1}
I20260619 18:53:45.026226 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.931317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:43887 (local address 127.15.91.3:54375)
0619 18:53:44.989585 (+ 58268us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:44.989616 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:44.996460 (+ 6844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:44.997167 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:44.997173 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:44.997853 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:44.998246 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:44.998259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.002308 (+ 4049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.002315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.004081 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.004103 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.005468 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.005479 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.007325 (+ 1846us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.008053 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:45.008088 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:45.012222 (+ 4134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:45.017779 (+ 5557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:45.017788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:45.017795 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:45.018566 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:45.019112 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:45.019117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:45.019120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:45.019304 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:45.020138 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:45.020146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:45.020315 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 18:53:45.025361 (+ 5046us) client_negotiation.cc:240] Negotiation successful
0619 18:53:45.025752 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58009,"mutex_wait_us":437,"spinlock_wait_cycles":5504,"thread_start_us":385,"threads_started":1}
I20260619 18:53:45.027787 23160 heartbeater.cc:347] Connected to a master server at 127.15.91.60:43887
I20260619 18:53:45.027987 23160 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:45.028544 23160 heartbeater.cc:511] Master 127.15.91.60:43887 requested a full tablet report, sending...
I20260619 18:53:45.030012 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.933777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:33167 (local address 127.15.91.61:35667)
0619 18:53:44.971243 (+ 37466us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:44.971253 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:44.997056 (+ 25803us) server_negotiation.cc:407] Connection header received
0619 18:53:45.001716 (+ 4660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:45.001725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:45.001824 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:45.001960 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:45.005734 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.007069 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.009949 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.010674 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.010863 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.020673 (+ 9810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:45.020682 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:45.020688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:45.020733 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:45.025102 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.027320 (+ 2218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.027328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.027337 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.027567 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.028315 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.028323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.028328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.028727 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:45.028891 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.029413 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.029696 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37291,"thread_start_us":37172,"threads_started":1}
I20260619 18:53:45.030061 22691 ts_manager.cc:194] Registered new tserver with Master: a60a1830734746bfa8ab432167bf9cef (127.15.91.3:38517)
I20260619 18:53:45.031270 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:44.933677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35667 (local address 127.15.91.3:33167)
0619 18:53:44.995098 (+ 61421us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:44.995123 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:45.001305 (+ 6182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:45.004472 (+ 3167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:45.004478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:45.005145 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:45.005546 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.005557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.007836 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.007845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.009758 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.009775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.012756 (+ 2981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.012763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.014262 (+ 1499us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.015392 (+ 1130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:45.015424 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:45.020427 (+ 5003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:45.026491 (+ 6064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:45.026499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:45.026504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:45.027140 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:45.027776 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:45.027783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:45.027788 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:45.028140 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:45.028899 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:45.028904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:45.029118 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 18:53:45.029246 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 18:53:45.030795 (+ 1549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61228,"spinlock_wait_cycles":32000,"thread_start_us":55479,"threads_started":1}
I20260619 18:53:45.032213 23162 heartbeater.cc:347] Connected to a master server at 127.15.91.61:35667
I20260619 18:53:45.032418 23162 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:45.032778 23162 heartbeater.cc:511] Master 127.15.91.61:35667 requested a full tablet report, sending...
I20260619 18:53:45.034057 22501 ts_manager.cc:194] Registered new tserver with Master: a60a1830734746bfa8ab432167bf9cef (127.15.91.3:38517)
I20260619 18:53:45.030411 21900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:53:45.038937 21900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:45.104331 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.077402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:39293 (local address 127.0.0.1:59868)
0619 18:53:45.093390 (+ 15988us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:45.093415 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:45.093629 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:45.096261 (+ 2632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:45.096268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:45.096303 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:45.096683 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.096696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.098853 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.098860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.101541 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.101559 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.103275 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.103282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.103514 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.103559 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 18:53:45.103651 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 18:53:45.103826 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15815,"mutex_wait_us":1,"thread_start_us":17503,"threads_started":1}
I20260619 18:53:45.105820 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.077879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:38517 (local address 127.0.0.1:36036)
0619 18:53:45.095222 (+ 17343us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:45.095251 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:45.095455 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:45.096604 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:45.096613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:45.096666 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:45.097184 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.097202 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.099366 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.099376 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.102838 (+ 3462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.102867 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.105151 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.105159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.105319 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.105349 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:53:45.105440 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 18:53:45.105556 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17159,"thread_start_us":15160,"threads_started":1}
I20260619 18:53:45.105866 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.077667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.15.91.2:39293)
0619 18:53:45.094276 (+ 16609us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:45.094291 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:45.094338 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:53:45.094479 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:45.094490 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:45.095984 (+ 1494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:45.096275 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:45.096908 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.098581 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.101765 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.102999 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.104913 (+ 1914us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.105214 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.105308 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.105536 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16357,"thread_start_us":16215,"threads_started":1}
I20260619 18:53:45.106066 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.078139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.15.91.3:38517)
0619 18:53:45.092500 (+ 14361us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:45.092515 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:45.095415 (+ 2900us) server_negotiation.cc:407] Connection header received
0619 18:53:45.095644 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:45.095649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:45.095937 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:45.096207 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:45.097436 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.099192 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.103078 (+ 3886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.104925 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.105075 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.105253 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.105543 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.105718 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14035,"thread_start_us":14383,"threads_started":1}
I20260619 18:53:45.110335 22953 tablet_service.cc:1467] Processing CreateTablet for tablet 006e8a7ab5754d72bac78dd563514319 (DEFAULT_TABLE table=test-table [id=cef556799d16432e8aac61fc0b9aae59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:45.110342 23095 tablet_service.cc:1467] Processing CreateTablet for tablet 006e8a7ab5754d72bac78dd563514319 (DEFAULT_TABLE table=test-table [id=cef556799d16432e8aac61fc0b9aae59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:45.111907 22953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006e8a7ab5754d72bac78dd563514319. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:45.111948 23095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006e8a7ab5754d72bac78dd563514319. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:45.115785 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.097613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.0.0.1:41332)
0619 18:53:45.103746 (+ 6133us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:45.103775 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:45.103961 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:45.106928 (+ 2967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:45.106935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:45.106969 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:45.108026 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.108045 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.110098 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.110104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.112270 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.112303 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.114810 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.114819 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.115052 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.115094 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 18:53:45.115223 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 18:53:45.115396 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5884,"thread_start_us":6494,"threads_started":1}
I20260619 18:53:45.115886 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.093967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.15.91.1:35739)
0619 18:53:45.104887 (+ 10920us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:45.104898 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:45.104935 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:53:45.106354 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:45.106366 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:45.106632 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:45.106947 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:45.108316 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.109863 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.112532 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.114168 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.114343 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.114572 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.115326 (+ 754us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.115544 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10717,"thread_start_us":10913,"threads_started":1}
I20260619 18:53:45.120214 22805 tablet_service.cc:1467] Processing CreateTablet for tablet 006e8a7ab5754d72bac78dd563514319 (DEFAULT_TABLE table=test-table [id=cef556799d16432e8aac61fc0b9aae59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:45.121834 22805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006e8a7ab5754d72bac78dd563514319. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:45.126971 23197 tablet_bootstrap.cc:492] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: Bootstrap starting.
I20260619 18:53:45.131742 23197 tablet_bootstrap.cc:654] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:45.133169 23197 log.cc:824] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:45.133641 23198 tablet_bootstrap.cc:492] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: Bootstrap starting.
I20260619 18:53:45.137713 23197 tablet_bootstrap.cc:492] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: No bootstrap required, opened a new log
I20260619 18:53:45.137796 23198 tablet_bootstrap.cc:654] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:45.138072 23197 ts_tablet_manager.cc:1397] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 18:53:45.138840 23200 tablet_bootstrap.cc:492] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: Bootstrap starting.
I20260619 18:53:45.139300 23198 log.cc:824] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:45.142776 23200 tablet_bootstrap.cc:654] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:45.144011 23200 log.cc:824] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:45.147511 23198 tablet_bootstrap.cc:492] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: No bootstrap required, opened a new log
I20260619 18:53:45.147696 23200 tablet_bootstrap.cc:492] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: No bootstrap required, opened a new log
I20260619 18:53:45.147917 23198 ts_tablet_manager.cc:1397] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260619 18:53:45.148015 23200 ts_tablet_manager.cc:1397] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 18:53:45.151399 23197 raft_consensus.cc:348] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } } peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } }
I20260619 18:53:45.151995 23197 raft_consensus.cc:374] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:45.152112 23197 raft_consensus.cc:729] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60a1830734746bfa8ab432167bf9cef, State: Initialized, Role: FOLLOWER
I20260619 18:53:45.152930 23197 consensus_queue.cc:260] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } } peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } }
I20260619 18:53:45.155098 23197 ts_tablet_manager.cc:1428] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: Time spent starting tablet: real 0.017s user 0.019s sys 0.000s
I20260619 18:53:45.155447 23161 heartbeater.cc:503] Master 127.15.91.62:34269 was elected leader, sending a full tablet report...
I20260619 18:53:45.158383 23198 raft_consensus.cc:348] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } } peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } }
I20260619 18:53:45.158949 23198 raft_consensus.cc:374] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:45.159071 23198 raft_consensus.cc:729] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb4667c5c934fd090783186dd64a3a2, State: Initialized, Role: FOLLOWER
I20260619 18:53:45.159791 23198 consensus_queue.cc:260] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } } peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } }
I20260619 18:53:45.160755 23200 raft_consensus.cc:348] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } } peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } }
I20260619 18:53:45.161432 23200 raft_consensus.cc:374] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:45.161619 23200 raft_consensus.cc:729] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f612cc551a4147868957b40b15baf8, State: Initialized, Role: FOLLOWER
I20260619 18:53:45.162345 23200 consensus_queue.cc:260] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } } peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } }
I20260619 18:53:45.163143 23198 ts_tablet_manager.cc:1428] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260619 18:53:45.163486 23019 heartbeater.cc:503] Master 127.15.91.62:34269 was elected leader, sending a full tablet report...
I20260619 18:53:45.166970 23200 ts_tablet_manager.cc:1428] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
W20260619 18:53:45.186987 23163 tablet.cc:2369] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:53:45.221820 23203 raft_consensus.cc:482] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:45.222198 23203 raft_consensus.cc:504] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } } peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } }
I20260619 18:53:45.224187 23203 leader_election.cc:288] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebb4667c5c934fd090783186dd64a3a2 (127.15.91.2:39293), d1f612cc551a4147868957b40b15baf8 (127.15.91.1:35739)
I20260619 18:53:45.232697 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.224632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:45785 (local address 127.15.91.1:35739)
0619 18:53:45.224843 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:45.224851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:45.224876 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:53:45.225541 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:45.225545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:45.225614 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:45.225727 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:45.226083 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.227131 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.229977 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.231379 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.231544 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.231663 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.232290 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.232453 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:53:45.232712 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.224502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.15.91.3:45785)
0619 18:53:45.224807 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:45.224824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:45.224938 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:45.225767 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:45.225770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:45.225789 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:45.225934 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.225940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.227290 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.227295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.229694 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.229718 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.231639 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.231647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.231966 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.232097 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 18:53:45.232217 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 18:53:45.232376 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":343,"spinlock_wait_cycles":9984}
I20260619 18:53:45.232808 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.224949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:36825 (local address 127.15.91.2:39293)
0619 18:53:45.225136 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:45.225140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:45.225155 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:53:45.225219 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:45.225223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:45.225279 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:45.225360 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:45.225749 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.226922 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.229977 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.231393 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.231544 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.231670 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.232350 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.232561 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:53:45.233364 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.224502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:39293 (local address 127.15.91.3:36825)
0619 18:53:45.224757 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:45.224778 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:45.224938 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:45.225400 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:45.225403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:45.225430 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:45.225589 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.225596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.227115 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.227120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.229694 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:45.229718 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.231639 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.231647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:45.231964 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.232097 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 18:53:45.232207 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 18:53:45.233073 (+ 866us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":289,"spinlock_wait_cycles":31488}
I20260619 18:53:45.234355 22825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006e8a7ab5754d72bac78dd563514319" candidate_uuid: "a60a1830734746bfa8ab432167bf9cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f612cc551a4147868957b40b15baf8" is_pre_election: true
I20260619 18:53:45.234462 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006e8a7ab5754d72bac78dd563514319" candidate_uuid: "a60a1830734746bfa8ab432167bf9cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb4667c5c934fd090783186dd64a3a2" is_pre_election: true
I20260619 18:53:45.235081 22973 raft_consensus.cc:2454] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60a1830734746bfa8ab432167bf9cef in term 0.
I20260619 18:53:45.235081 22825 raft_consensus.cc:2454] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60a1830734746bfa8ab432167bf9cef in term 0.
I20260619 18:53:45.235982 23051 leader_election.cc:302] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a60a1830734746bfa8ab432167bf9cef, d1f612cc551a4147868957b40b15baf8, ebb4667c5c934fd090783186dd64a3a2; no voters:
I20260619 18:53:45.236485 23203 raft_consensus.cc:2781] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:45.236665 23203 raft_consensus.cc:482] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:45.236779 23203 raft_consensus.cc:3037] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:45.239357 23203 raft_consensus.cc:504] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } } peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } }
I20260619 18:53:45.240151 23203 leader_election.cc:288] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [CANDIDATE]: Term 1 election: Requested vote from peers ebb4667c5c934fd090783186dd64a3a2 (127.15.91.2:39293), d1f612cc551a4147868957b40b15baf8 (127.15.91.1:35739)
I20260619 18:53:45.240748 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006e8a7ab5754d72bac78dd563514319" candidate_uuid: "a60a1830734746bfa8ab432167bf9cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb4667c5c934fd090783186dd64a3a2"
I20260619 18:53:45.240811 22825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006e8a7ab5754d72bac78dd563514319" candidate_uuid: "a60a1830734746bfa8ab432167bf9cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f612cc551a4147868957b40b15baf8"
I20260619 18:53:45.241017 22973 raft_consensus.cc:3037] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:45.241027 22825 raft_consensus.cc:3037] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:45.244578 22825 raft_consensus.cc:2454] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60a1830734746bfa8ab432167bf9cef in term 1.
I20260619 18:53:45.244578 22973 raft_consensus.cc:2454] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60a1830734746bfa8ab432167bf9cef in term 1.
I20260619 18:53:45.245193 23051 leader_election.cc:302] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60a1830734746bfa8ab432167bf9cef, ebb4667c5c934fd090783186dd64a3a2; no voters:
I20260619 18:53:45.245731 23203 raft_consensus.cc:2781] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:45.246291 23203 raft_consensus.cc:686] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 1 LEADER]: Becoming Leader. State: Replica: a60a1830734746bfa8ab432167bf9cef, State: Running, Role: LEADER
I20260619 18:53:45.246933 23203 consensus_queue.cc:237] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } } peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } }
I20260619 18:53:45.253146 21900 catalog_manager.cc:5495] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef reported cstate change: term changed from 0 to 1, leader changed from <none> to a60a1830734746bfa8ab432167bf9cef (127.15.91.3). New cstate: current_term: 1 leader_uuid: "a60a1830734746bfa8ab432167bf9cef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } health_report { overall_health: UNKNOWN } } }
W20260619 18:53:45.278841 23021 tablet.cc:2369] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:53:45.285970 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.279169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.15.91.3:38517)
0619 18:53:45.279394 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:45.279399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:45.279417 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:53:45.279516 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:45.279520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:45.279668 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:45.279806 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:45.280172 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.281028 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.282804 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.283397 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.283522 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.284168 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:45.284758 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:45.284909 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.285415 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.285544 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:53:45.296295 22825 raft_consensus.cc:1261] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8 [term 1 FOLLOWER]: Refusing update from remote peer a60a1830734746bfa8ab432167bf9cef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:45.296335 22973 raft_consensus.cc:1261] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2 [term 1 FOLLOWER]: Refusing update from remote peer a60a1830734746bfa8ab432167bf9cef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:45.297567 23206 consensus_queue.cc:1035] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:45.297976 23203 consensus_queue.cc:1035] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:53:45.313876 22873 tablet.cc:2369] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:53:45.329932 23212 mvcc.cc:205] Tried to move back new op lower bound from 7298642842801000448 to 7298642842616446976. Current Snapshot: MvccSnapshot[applied={T|T < 7298642842801000448 or (T in {7298642842801000448})}]
I20260619 18:53:45.337241 21900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:45.337554 21900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:45.345005 21900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68bcf1108b5f49d398f22fe37bc8c056: Sending DeleteTablet for 3 replicas of tablet 006e8a7ab5754d72bac78dd563514319
I20260619 18:53:45.346379 22953 tablet_service.cc:1514] Processing DeleteTablet for tablet 006e8a7ab5754d72bac78dd563514319 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:45 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:59868
I20260619 18:53:45.347494 23219 tablet_replica.cc:330] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: stopping tablet replica
I20260619 18:53:45.348272 23095 tablet_service.cc:1514] Processing DeleteTablet for tablet 006e8a7ab5754d72bac78dd563514319 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:45 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:36036
I20260619 18:53:45.348263 22805 tablet_service.cc:1514] Processing DeleteTablet for tablet 006e8a7ab5754d72bac78dd563514319 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:45 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:41332
I20260619 18:53:45.349233 23220 tablet_replica.cc:330] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: stopping tablet replica
I20260619 18:53:45.349781 23220 raft_consensus.cc:2229] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:45.350525 23220 raft_consensus.cc:2258] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:45.354399 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 21868
I20260619 18:53:45.354599 23219 raft_consensus.cc:2229] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:45.355250 23219 raft_consensus.cc:2258] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:45.355847 21930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:45.358312 23219 ts_tablet_manager.cc:1905] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:45.361053 23221 tablet_replica.cc:330] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: stopping tablet replica
I20260619 18:53:45.361842 23221 raft_consensus.cc:2229] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:45.362463 23221 raft_consensus.cc:2258] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:45.364411 23220 ts_tablet_manager.cc:1905] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:45.364845 23221 ts_tablet_manager.cc:1905] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:45.365144 23219 ts_tablet_manager.cc:1918] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:45.365345 23219 log.cc:1196] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/006e8a7ab5754d72bac78dd563514319
I20260619 18:53:45.365980 23219 ts_tablet_manager.cc:1939] T 006e8a7ab5754d72bac78dd563514319 P ebb4667c5c934fd090783186dd64a3a2: Deleting consensus metadata
I20260619 18:53:45.373401 23221 ts_tablet_manager.cc:1918] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:45.373730 23221 log.cc:1196] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/006e8a7ab5754d72bac78dd563514319
I20260619 18:53:45.374508 23221 ts_tablet_manager.cc:1939] T 006e8a7ab5754d72bac78dd563514319 P d1f612cc551a4147868957b40b15baf8: Deleting consensus metadata
I20260619 18:53:45.374944 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.351782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.15.91.61:35667)
0619 18:53:45.351971 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:45.351977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:45.352011 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:53:45.352079 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:45.352084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:45.352201 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:45.352345 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:45.354522 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.355584 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.357670 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.358356 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.358483 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.367433 (+ 8950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:45.367442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:45.367447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:45.367517 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:45.370681 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.371802 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.371809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.371813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.371980 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.372456 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.372465 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.372471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.372913 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:45.373162 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.374073 (+ 911us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.374387 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:53:45.383306 23220 ts_tablet_manager.cc:1918] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:45.383598 23220 log.cc:1196] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/006e8a7ab5754d72bac78dd563514319
I20260619 18:53:45.385219 23220 ts_tablet_manager.cc:1939] T 006e8a7ab5754d72bac78dd563514319 P a60a1830734746bfa8ab432167bf9cef: Deleting consensus metadata
I20260619 18:53:45.395998 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.352742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.15.91.60:43887)
0619 18:53:45.352950 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:45.352957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:45.376288 (+ 23331us) server_negotiation.cc:407] Connection header received
0619 18:53:45.376496 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:45.376502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:45.376591 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:45.376694 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:45.379297 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.380237 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.382031 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.382909 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.383074 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.388042 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:45.388050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:45.388056 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:45.388109 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:45.392179 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.393283 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.393291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.393294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.393480 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.394059 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.394065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.394067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.394528 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:45.394889 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.395257 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.395573 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:53:45.499760 21884 catalog_manager.cc:4841] TS a60a1830734746bfa8ab432167bf9cef (127.15.91.3:38517): tablet 006e8a7ab5754d72bac78dd563514319 (table test-table [id=cef556799d16432e8aac61fc0b9aae59]) successfully deleted
I20260619 18:53:45.501644 21887 catalog_manager.cc:4841] TS ebb4667c5c934fd090783186dd64a3a2 (127.15.91.2:39293): tablet 006e8a7ab5754d72bac78dd563514319 (table test-table [id=cef556799d16432e8aac61fc0b9aae59]) successfully deleted
I20260619 18:53:45.502022 21884 catalog_manager.cc:4841] TS d1f612cc551a4147868957b40b15baf8 (127.15.91.1:35739): tablet 006e8a7ab5754d72bac78dd563514319 (table test-table [id=cef556799d16432e8aac61fc0b9aae59]) successfully deleted
W20260619 18:53:45.501945 21887 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.15.91.2:39293, user_credentials={real_user=slave}} blocked reactor thread for 133743us
I20260619 18:53:45.502684 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 21868
I20260619 18:53:45.562376 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.542168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.15.91.60:43887)
0619 18:53:45.542379 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:45.542384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:45.542856 (+ 472us) server_negotiation.cc:407] Connection header received
0619 18:53:45.543091 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:45.543099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:45.543186 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:45.543294 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:45.545357 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.546590 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.548897 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.549681 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.549831 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.553599 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:45.553607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:45.553612 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:45.553662 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:45.558020 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.559398 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.559405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.559410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.559618 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.560193 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.560199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.560203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.560639 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:45.560920 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.561650 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.561954 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 18:53:45.563570 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:45.540727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.15.91.61:35667)
0619 18:53:45.540928 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:45.540936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:45.542048 (+ 1112us) server_negotiation.cc:407] Connection header received
0619 18:53:45.542567 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:45.542574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:45.542670 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:45.542828 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:45.545387 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.546164 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.548474 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:45.548928 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:45.549107 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:45.552997 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:45.553004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:45.553007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:45.553055 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:45.558014 (+ 4959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.560601 (+ 2587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.560607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.560611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.560788 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:45.561364 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:45.561371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:45.561375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:45.561772 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:45.562060 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:45.562935 (+ 875us) server_negotiation.cc:299] Negotiation successful
0619 18:53:45.563202 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:53:46.323724 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.322708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.3:54551)
0619 18:53:46.323325 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.323476 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":332,"threads_started":1}
W20260619 18:53:46.324177 23161 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:34269 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
I20260619 18:53:46.331543 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.330576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.2:35165)
0619 18:53:46.331131 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.331318 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":286,"threads_started":1}
W20260619 18:53:46.332043 23019 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:34269 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
I20260619 18:53:46.335642 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.334689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.1:44851)
0619 18:53:46.335312 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.335443 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":355,"threads_started":1}
W20260619 18:53:46.336062 22871 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:34269 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
I20260619 18:53:46.845312 23233 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68bcf1108b5f49d398f22fe37bc8c056)
I20260619 18:53:46.845526 23233 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } }
I20260619 18:53:46.846356 23233 leader_election.cc:288] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68bcf1108b5f49d398f22fe37bc8c056 (127.15.91.62:34269), 3943f18ae29c4d5a99391d6b77d60a8e (127.15.91.61:35667)
I20260619 18:53:46.859417 23237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68bcf1108b5f49d398f22fe37bc8c056)
I20260619 18:53:46.859949 23237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } }
I20260619 18:53:46.861598 23237 leader_election.cc:288] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68bcf1108b5f49d398f22fe37bc8c056 (127.15.91.62:34269), ccea9e7e30e34087952516eb8a4ac28b (127.15.91.60:43887)
I20260619 18:53:46.863636 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.862038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:49922)
0619 18:53:46.863011 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.863252 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":552,"threads_started":1}
I20260619 18:53:46.863981 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.846615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:49916)
0619 18:53:46.863296 (+ 16681us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.863537 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16567,"mutex_wait_us":36,"threads_started":1}
W20260619 18:53:46.864195 22488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
W20260619 18:53:46.864534 22678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
I20260619 18:53:46.871531 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.870871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:49930)
0619 18:53:46.871212 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.871330 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260619 18:53:46.871809 22678 leader_election.cc:334] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68bcf1108b5f49d398f22fe37bc8c056 (127.15.91.62:34269): Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
I20260619 18:53:46.873932 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.846615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35667 (local address 127.0.0.1:52618)
0619 18:53:46.862571 (+ 15956us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.862604 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:46.862849 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:46.864627 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:46.864632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:46.864661 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:46.865021 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:46.865039 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.868293 (+ 3254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.868303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:46.870848 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:46.870865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.872989 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.872995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:46.873178 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:46.873222 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 18:53:46.873349 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 18:53:46.873532 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15737,"thread_start_us":16339,"threads_started":1}
I20260619 18:53:46.874325 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.846843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.15.91.61:35667)
0619 18:53:46.862444 (+ 15601us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:46.862455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:46.862760 (+ 305us) server_negotiation.cc:407] Connection header received
0619 18:53:46.864044 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:46.864053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:46.864182 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:46.864277 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:46.865963 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.867836 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.871053 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.872526 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.872659 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:46.872778 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:46.873436 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 18:53:46.873696 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15409,"thread_start_us":15201,"threads_started":1}
I20260619 18:53:46.875124 22511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccea9e7e30e34087952516eb8a4ac28b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" is_pre_election: true
I20260619 18:53:46.875514 22511 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccea9e7e30e34087952516eb8a4ac28b in term 1.
I20260619 18:53:46.876219 22675 leader_election.cc:302] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3943f18ae29c4d5a99391d6b77d60a8e, ccea9e7e30e34087952516eb8a4ac28b; no voters: 68bcf1108b5f49d398f22fe37bc8c056
I20260619 18:53:46.876510 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.875754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:49936)
0619 18:53:46.876027 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.876219 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 18:53:46.876837 22488 leader_election.cc:334] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68bcf1108b5f49d398f22fe37bc8c056 (127.15.91.62:34269): Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
I20260619 18:53:46.876853 23233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 18:53:46.877034 23233 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 68bcf1108b5f49d398f22fe37bc8c056)
I20260619 18:53:46.877152 23233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:53:46.878379 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.862899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:43887 (local address 127.0.0.1:60526)
0619 18:53:46.868895 (+ 5996us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.868931 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:46.869163 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:46.870195 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:46.870200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:46.870227 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:46.870557 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:46.870570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.873477 (+ 2907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.873482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:46.875782 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:46.875798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.877678 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.877684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:46.877839 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:46.877867 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:53:46.877973 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 18:53:46.878127 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5786,"thread_start_us":6541,"threads_started":1}
I20260619 18:53:46.878635 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.863317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.15.91.60:43887)
0619 18:53:46.868020 (+ 4703us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:46.868027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:46.869874 (+ 1847us) server_negotiation.cc:407] Connection header received
0619 18:53:46.869961 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:46.869966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:46.870051 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:46.870186 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:46.872147 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.873288 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.876027 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.877436 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.877619 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:46.877757 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:46.878030 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 18:53:46.878203 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4492,"thread_start_us":4687,"threads_started":1}
I20260619 18:53:46.879333 22701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ccea9e7e30e34087952516eb8a4ac28b" is_pre_election: true
I20260619 18:53:46.880335 23233 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } }
I20260619 18:53:46.880831 22701 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3943f18ae29c4d5a99391d6b77d60a8e in current term 2: Already voted for candidate ccea9e7e30e34087952516eb8a4ac28b in this term.
I20260619 18:53:46.881108 23233 leader_election.cc:288] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [CANDIDATE]: Term 2 election: Requested vote from peers 68bcf1108b5f49d398f22fe37bc8c056 (127.15.91.62:34269), 3943f18ae29c4d5a99391d6b77d60a8e (127.15.91.61:35667)
I20260619 18:53:46.881740 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.881267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:49940)
0619 18:53:46.881453 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.881552 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 18:53:46.881785 22511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccea9e7e30e34087952516eb8a4ac28b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3943f18ae29c4d5a99391d6b77d60a8e"
I20260619 18:53:46.882054 22511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:53:46.882498 22485 leader_election.cc:302] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3943f18ae29c4d5a99391d6b77d60a8e; no voters: 68bcf1108b5f49d398f22fe37bc8c056, ccea9e7e30e34087952516eb8a4ac28b
I20260619 18:53:46.883054 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.882559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:49950)
0619 18:53:46.882770 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.882885 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 18:53:46.883301 22678 leader_election.cc:334] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 68bcf1108b5f49d398f22fe37bc8c056 (127.15.91.62:34269): Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
I20260619 18:53:46.884974 22511 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccea9e7e30e34087952516eb8a4ac28b in term 2.
I20260619 18:53:46.885344 23237 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:53:46.885533 22675 leader_election.cc:302] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3943f18ae29c4d5a99391d6b77d60a8e, ccea9e7e30e34087952516eb8a4ac28b; no voters: 68bcf1108b5f49d398f22fe37bc8c056
I20260619 18:53:46.885917 23233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:53:46.886457 23233 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [term 2 LEADER]: Becoming Leader. State: Replica: ccea9e7e30e34087952516eb8a4ac28b, State: Running, Role: LEADER
I20260619 18:53:46.886952 23233 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } }
I20260619 18:53:46.889379 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccea9e7e30e34087952516eb8a4ac28b. Latest consensus state: current_term: 2 leader_uuid: "ccea9e7e30e34087952516eb8a4ac28b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:46.889861 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:46.890640 23245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:46.894899 23245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:46.895927 23245 catalog_manager.cc:1259] Loaded cluster ID: 40b587c88d5540ceb49e34cfd5421bbf
I20260619 18:53:46.896018 23245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:46.897274 23245 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:46.898167 23245 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Loaded TSK: 0
I20260619 18:53:46.899221 23245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:53:46.928472 22690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:53:46.936239 22690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:46.941269 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.940691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:49964)
0619 18:53:46.940947 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.941060 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 18:53:46.941274 22511 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [term 2 FOLLOWER]: Refusing update from remote peer ccea9e7e30e34087952516eb8a4ac28b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 18:53:46.942173 23243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 18:53:46.942551 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.942077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:49968)
0619 18:53:46.942276 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.942359 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 18:53:46.942787 22678 consensus_peers.cc:487] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b -> Peer 68bcf1108b5f49d398f22fe37bc8c056 (127.15.91.62:34269): Couldn't send request to peer 68bcf1108b5f49d398f22fe37bc8c056. Status: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 18:53:46.946352 23237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccea9e7e30e34087952516eb8a4ac28b. Latest consensus state: current_term: 2 leader_uuid: "ccea9e7e30e34087952516eb8a4ac28b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:46.946813 23237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:53:46.948776 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ccea9e7e30e34087952516eb8a4ac28b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:46.949134 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:46.950563 23237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ccea9e7e30e34087952516eb8a4ac28b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "68bcf1108b5f49d398f22fe37bc8c056" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34269 } } peers { permanent_uuid: "3943f18ae29c4d5a99391d6b77d60a8e" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35667 } attrs { promote: false } } peers { permanent_uuid: "ccea9e7e30e34087952516eb8a4ac28b" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 43887 } attrs { promote: false } } }
I20260619 18:53:46.950929 23237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3943f18ae29c4d5a99391d6b77d60a8e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:53:46.978331 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.960538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.0.0.1:41340)
0619 18:53:46.960889 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.960907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:46.961026 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:46.972261 (+ 11235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:46.972266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:46.972314 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:46.972628 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:46.972638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.973828 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.973835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:46.975519 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:46.975541 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.977393 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.977400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:46.977640 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:46.977678 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 18:53:46.977775 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 18:53:46.977947 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260619 18:53:46.979300 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.960899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.15.91.1:35739)
0619 18:53:46.971885 (+ 10986us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:46.971890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:46.971912 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:53:46.972019 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:46.972025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:46.972106 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:46.972235 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:46.972815 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.973659 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.975784 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.977088 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.977265 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:46.977400 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:46.978774 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0619 18:53:46.978955 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10851,"thread_start_us":10725,"threads_started":1}
I20260619 18:53:46.980224 22805 tablet_service.cc:1467] Processing CreateTablet for tablet 69518ff63dff4fc1b15ff1784a9749b7 (DEFAULT_TABLE table=test-table [id=de9af4cf5a39460d8c22804a2fa86361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:46.980798 22805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69518ff63dff4fc1b15ff1784a9749b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:46.982650 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.960220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:38517 (local address 127.0.0.1:36046)
0619 18:53:46.960376 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.960399 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:46.960514 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:46.976018 (+ 15504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:46.976024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:46.976064 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:46.976257 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:46.976272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.978044 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.978049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:46.980059 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:46.980076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.981790 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.981797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:46.982019 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:46.982047 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:53:46.982176 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 18:53:46.982340 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":4}
I20260619 18:53:46.982688 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.960340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.15.91.3:38517)
0619 18:53:46.975670 (+ 15330us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:46.975674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:46.975695 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:53:46.975788 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:46.975791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:46.975865 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:46.975963 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:46.976490 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.977857 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.980402 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.981591 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.981762 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:46.981875 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:46.982210 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 18:53:46.982366 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15190,"thread_start_us":15078,"threads_started":1}
I20260619 18:53:46.983593 23095 tablet_service.cc:1467] Processing CreateTablet for tablet 69518ff63dff4fc1b15ff1784a9749b7 (DEFAULT_TABLE table=test-table [id=de9af4cf5a39460d8c22804a2fa86361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:46.983940 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.960220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:39293 (local address 127.0.0.1:59870)
0619 18:53:46.970791 (+ 10571us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:46.970840 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:46.971081 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:46.976808 (+ 5727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:46.976814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:46.976844 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:46.977260 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:46.977274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.978892 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.978897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:46.981019 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:46.981039 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.982988 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.982995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:46.983222 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:46.983261 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 18:53:46.983352 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 18:53:46.983513 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10317,"mutex_wait_us":892,"thread_start_us":10048,"threads_started":1}
I20260619 18:53:46.984138 23095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69518ff63dff4fc1b15ff1784a9749b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:46.984946 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:46.960361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59870 (local address 127.15.91.2:39293)
0619 18:53:46.976444 (+ 16083us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:46.976459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:46.976478 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:53:46.976554 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:46.976560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:46.976646 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:46.976764 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:46.977524 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.978581 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.981384 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:46.982662 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:46.984281 (+ 1619us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:46.984420 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:46.984488 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 18:53:46.984651 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15909,"thread_start_us":16521,"threads_started":1}
I20260619 18:53:46.987298 22953 tablet_service.cc:1467] Processing CreateTablet for tablet 69518ff63dff4fc1b15ff1784a9749b7 (DEFAULT_TABLE table=test-table [id=de9af4cf5a39460d8c22804a2fa86361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:46.987879 22953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69518ff63dff4fc1b15ff1784a9749b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:46.989365 23255 tablet_bootstrap.cc:492] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8: Bootstrap starting.
I20260619 18:53:46.991287 23256 tablet_bootstrap.cc:492] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef: Bootstrap starting.
I20260619 18:53:46.991632 23255 tablet_bootstrap.cc:654] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:46.993891 23256 tablet_bootstrap.cc:654] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:46.996906 23256 tablet_bootstrap.cc:492] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef: No bootstrap required, opened a new log
I20260619 18:53:46.997108 23256 ts_tablet_manager.cc:1397] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 18:53:46.997143 23255 tablet_bootstrap.cc:492] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8: No bootstrap required, opened a new log
I20260619 18:53:46.997290 23255 ts_tablet_manager.cc:1397] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 18:53:46.998088 23256 raft_consensus.cc:348] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } } peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } }
I20260619 18:53:46.998090 23255 raft_consensus.cc:348] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } } peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } }
I20260619 18:53:46.998370 23256 raft_consensus.cc:374] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:46.998378 23255 raft_consensus.cc:374] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:46.998479 23256 raft_consensus.cc:729] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60a1830734746bfa8ab432167bf9cef, State: Initialized, Role: FOLLOWER
I20260619 18:53:46.998479 23255 raft_consensus.cc:729] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f612cc551a4147868957b40b15baf8, State: Initialized, Role: FOLLOWER
I20260619 18:53:46.998804 23255 consensus_queue.cc:260] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } } peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } }
I20260619 18:53:46.998893 23256 consensus_queue.cc:260] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } } peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } }
I20260619 18:53:46.999979 23255 ts_tablet_manager.cc:1428] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 18:53:47.002240 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.000889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.1:38967)
0619 18:53:47.001606 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.001823 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":338,"threads_started":1}
I20260619 18:53:47.002333 23258 tablet_bootstrap.cc:492] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2: Bootstrap starting.
I20260619 18:53:47.004611 23258 tablet_bootstrap.cc:654] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:47.004897 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.004123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.1:51057)
0619 18:53:47.004477 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.004602 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260619 18:53:47.008702 23258 tablet_bootstrap.cc:492] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2: No bootstrap required, opened a new log
I20260619 18:53:47.008929 23258 ts_tablet_manager.cc:1397] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 18:53:47.009829 23258 raft_consensus.cc:348] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } } peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } }
I20260619 18:53:47.010109 23258 raft_consensus.cc:374] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:47.010211 23258 raft_consensus.cc:729] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb4667c5c934fd090783186dd64a3a2, State: Initialized, Role: FOLLOWER
I20260619 18:53:47.010491 23258 consensus_queue.cc:260] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } } peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } }
I20260619 18:53:47.010962 22870 heartbeater.cc:503] Master 127.15.91.60:43887 was elected leader, sending a full tablet report...
I20260619 18:53:47.012928 23256 ts_tablet_manager.cc:1428] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef: Time spent starting tablet: real 0.016s user 0.002s sys 0.006s
I20260619 18:53:47.016288 23160 heartbeater.cc:503] Master 127.15.91.60:43887 was elected leader, sending a full tablet report...
I20260619 18:53:47.016595 23258 ts_tablet_manager.cc:1428] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260619 18:53:47.020346 23018 heartbeater.cc:503] Master 127.15.91.60:43887 was elected leader, sending a full tablet report...
I20260619 18:53:47.020416 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.007044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.3:38561)
0619 18:53:47.019780 (+ 12736us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.020034 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12539,"threads_started":1}
I20260619 18:53:47.022228 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.021532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.3:42325)
0619 18:53:47.021862 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.021990 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260619 18:53:47.023568 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.017670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.2:58549)
0619 18:53:47.023193 (+ 5523us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.023322 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5395,"thread_start_us":5220,"threads_started":1}
I20260619 18:53:47.024678 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.024308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.2:46039)
0619 18:53:47.024480 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.024552 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 18:53:47.096377 23265 raft_consensus.cc:482] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:47.096630 23265 raft_consensus.cc:504] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } } peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } }
I20260619 18:53:47.098369 23265 leader_election.cc:288] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a60a1830734746bfa8ab432167bf9cef (127.15.91.3:38517), d1f612cc551a4147868957b40b15baf8 (127.15.91.1:35739)
I20260619 18:53:47.108227 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.099307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:32779 (local address 127.15.91.1:35739)
0619 18:53:47.099597 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:47.099604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:47.099926 (+ 322us) server_negotiation.cc:407] Connection header received
0619 18:53:47.100086 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:47.100090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:47.100152 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:47.100271 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:47.100782 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.101717 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.104940 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.106776 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.106973 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:47.107095 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:47.107755 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 18:53:47.107930 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 18:53:47.108234 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.099006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:52875 (local address 127.15.91.3:38517)
0619 18:53:47.099217 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:47.099226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:47.099251 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:53:47.099337 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:47.099345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:47.099452 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:47.099569 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:47.100275 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.101282 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.104940 (+ 3658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.106772 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.106981 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:47.107094 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:47.107755 (+ 661us) server_negotiation.cc:299] Negotiation successful
0619 18:53:47.107937 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:53:47.108283 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.098642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:38517 (local address 127.15.91.2:52875)
0619 18:53:47.098921 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.098942 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:47.099142 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:47.099705 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:47.099709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:47.099739 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:47.100112 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:47.100122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.101541 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.101547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:47.104547 (+ 3000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:47.104581 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.107151 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.107160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:47.107463 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:47.107539 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 18:53:47.107665 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:53:47.107892 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":146}
I20260619 18:53:47.108335 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.099143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.15.91.2:32779)
0619 18:53:47.099790 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.099813 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:47.099948 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:47.100309 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:47.100312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:47.100333 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:47.100615 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:47.100622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.101890 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.101895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:47.104547 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:47.104581 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.107151 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.107160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:47.107465 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:47.107539 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 18:53:47.107665 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:53:47.107884 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":175,"thread_start_us":347,"threads_started":1}
I20260619 18:53:47.109543 22825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69518ff63dff4fc1b15ff1784a9749b7" candidate_uuid: "ebb4667c5c934fd090783186dd64a3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f612cc551a4147868957b40b15baf8" is_pre_election: true
I20260619 18:53:47.109688 23115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69518ff63dff4fc1b15ff1784a9749b7" candidate_uuid: "ebb4667c5c934fd090783186dd64a3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60a1830734746bfa8ab432167bf9cef" is_pre_election: true
I20260619 18:53:47.109879 22825 raft_consensus.cc:2454] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb4667c5c934fd090783186dd64a3a2 in term 0.
I20260619 18:53:47.110270 23115 raft_consensus.cc:2454] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb4667c5c934fd090783186dd64a3a2 in term 0.
I20260619 18:53:47.110806 22906 leader_election.cc:302] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f612cc551a4147868957b40b15baf8, ebb4667c5c934fd090783186dd64a3a2; no voters:
I20260619 18:53:47.111263 23265 raft_consensus.cc:2781] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:47.111423 23265 raft_consensus.cc:482] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:47.111531 23265 raft_consensus.cc:3037] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:47.113837 23265 raft_consensus.cc:504] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } } peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } }
I20260619 18:53:47.114575 23265 leader_election.cc:288] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [CANDIDATE]: Term 1 election: Requested vote from peers a60a1830734746bfa8ab432167bf9cef (127.15.91.3:38517), d1f612cc551a4147868957b40b15baf8 (127.15.91.1:35739)
I20260619 18:53:47.115423 22825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69518ff63dff4fc1b15ff1784a9749b7" candidate_uuid: "ebb4667c5c934fd090783186dd64a3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f612cc551a4147868957b40b15baf8"
I20260619 18:53:47.115440 23115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69518ff63dff4fc1b15ff1784a9749b7" candidate_uuid: "ebb4667c5c934fd090783186dd64a3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60a1830734746bfa8ab432167bf9cef"
I20260619 18:53:47.115641 22825 raft_consensus.cc:3037] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:47.115806 23115 raft_consensus.cc:3037] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:47.118054 22825 raft_consensus.cc:2454] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb4667c5c934fd090783186dd64a3a2 in term 1.
I20260619 18:53:47.118785 22906 leader_election.cc:302] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f612cc551a4147868957b40b15baf8, ebb4667c5c934fd090783186dd64a3a2; no voters:
I20260619 18:53:47.118860 23115 raft_consensus.cc:2454] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb4667c5c934fd090783186dd64a3a2 in term 1.
I20260619 18:53:47.119302 23265 raft_consensus.cc:2781] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:47.119884 23265 raft_consensus.cc:686] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 1 LEADER]: Becoming Leader. State: Replica: ebb4667c5c934fd090783186dd64a3a2, State: Running, Role: LEADER
I20260619 18:53:47.120393 23265 consensus_queue.cc:237] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } } peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } }
I20260619 18:53:47.121328 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.120585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.2:36371)
0619 18:53:47.120983 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.121109 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260619 18:53:47.124626 22690 catalog_manager.cc:5495] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebb4667c5c934fd090783186dd64a3a2 (127.15.91.2). New cstate: current_term: 1 leader_uuid: "ebb4667c5c934fd090783186dd64a3a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb4667c5c934fd090783186dd64a3a2" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39293 } health_report { overall_health: HEALTHY } } }
I20260619 18:53:47.176542 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.170819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.15.91.2:39293)
0619 18:53:47.171058 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:47.171063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:47.171080 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:53:47.171176 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:47.171180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:47.171265 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:47.171446 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:47.171876 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.172602 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.174005 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.174519 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.174668 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:47.175185 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:47.175842 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:47.176009 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:47.176142 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 18:53:47.176235 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:53:47.181456 22825 raft_consensus.cc:1261] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8 [term 1 FOLLOWER]: Refusing update from remote peer ebb4667c5c934fd090783186dd64a3a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:47.181877 23115 raft_consensus.cc:1261] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef [term 1 FOLLOWER]: Refusing update from remote peer ebb4667c5c934fd090783186dd64a3a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:47.182348 23268 consensus_queue.cc:1035] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f612cc551a4147868957b40b15baf8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 35739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:47.182888 23265 consensus_queue.cc:1035] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60a1830734746bfa8ab432167bf9cef" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:47.185873 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.185259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.1:40437)
0619 18:53:47.185510 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.185644 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260619 18:53:47.187000 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.186386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.3:55677)
0619 18:53:47.186669 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.186800 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 18:53:47.193207 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.192702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.3:45229)
0619 18:53:47.192917 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.193027 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 18:53:47.201213 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.200597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.2:54419)
0619 18:53:47.200898 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.201008 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260619 18:53:47.206007 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.205426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.1:57529)
0619 18:53:47.205655 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.205765 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260619 18:53:47.208568 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.207986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.15.91.2:33155)
0619 18:53:47.208261 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.208382 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260619 18:53:47.215299 22691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:47.215667 22691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:47.223840 22691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccea9e7e30e34087952516eb8a4ac28b: Sending DeleteTablet for 3 replicas of tablet 69518ff63dff4fc1b15ff1784a9749b7
I20260619 18:53:47.225270 22805 tablet_service.cc:1514] Processing DeleteTablet for tablet 69518ff63dff4fc1b15ff1784a9749b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:47 UTC) from {username='oryx', principal='oryx/127.15.91.60@KRBTEST.COM'} at 127.0.0.1:41340
I20260619 18:53:47.225512 22953 tablet_service.cc:1514] Processing DeleteTablet for tablet 69518ff63dff4fc1b15ff1784a9749b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:47 UTC) from {username='oryx', principal='oryx/127.15.91.60@KRBTEST.COM'} at 127.0.0.1:59870
I20260619 18:53:47.226373 23095 tablet_service.cc:1514] Processing DeleteTablet for tablet 69518ff63dff4fc1b15ff1784a9749b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:47 UTC) from {username='oryx', principal='oryx/127.15.91.60@KRBTEST.COM'} at 127.0.0.1:36046
I20260619 18:53:47.230069 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 22663
I20260619 18:53:47.251264 23282 tablet_replica.cc:330] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2: stopping tablet replica
I20260619 18:53:47.251535 23282 raft_consensus.cc:2229] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:47.252197 23282 raft_consensus.cc:2258] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:47.252934 23281 tablet_replica.cc:330] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8: stopping tablet replica
I20260619 18:53:47.253172 23281 raft_consensus.cc:2229] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:47.253530 23281 raft_consensus.cc:2258] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:47.258915 23282 ts_tablet_manager.cc:1905] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:47.259548 23283 tablet_replica.cc:330] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef: stopping tablet replica
I20260619 18:53:47.259759 23283 raft_consensus.cc:2229] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:47.260187 23283 raft_consensus.cc:2258] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:47.261384 23281 ts_tablet_manager.cc:1905] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:47.264652 23283 ts_tablet_manager.cc:1905] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:47.269914 23282 ts_tablet_manager.cc:1918] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:47.270126 23282 log.cc:1196] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/69518ff63dff4fc1b15ff1784a9749b7
I20260619 18:53:47.277596 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.231696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.15.91.1:35739)
0619 18:53:47.232093 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:47.232102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:47.247933 (+ 15831us) server_negotiation.cc:407] Connection header received
0619 18:53:47.248532 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:47.248539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:47.248649 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:47.248722 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:47.250892 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.252051 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.256681 (+ 4630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.257403 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.257547 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:47.265799 (+ 8252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:47.265821 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:47.265854 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:47.265908 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:47.270425 (+ 4517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:47.271512 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:47.271522 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:47.271525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:47.271730 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:47.272227 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:47.272232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:47.272235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:47.272858 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:47.273189 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:47.276807 (+ 3618us) server_negotiation.cc:299] Negotiation successful
0619 18:53:47.277153 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 18:53:47.278527 22865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:47.283226 23282 ts_tablet_manager.cc:1939] T 69518ff63dff4fc1b15ff1784a9749b7 P ebb4667c5c934fd090783186dd64a3a2: Deleting consensus metadata
I20260619 18:53:47.284972 22678 catalog_manager.cc:4841] TS ebb4667c5c934fd090783186dd64a3a2 (127.15.91.2:39293): tablet 69518ff63dff4fc1b15ff1784a9749b7 (table test-table [id=de9af4cf5a39460d8c22804a2fa86361]) successfully deleted
I20260619 18:53:47.285946 23283 ts_tablet_manager.cc:1918] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:47.286283 23283 log.cc:1196] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/69518ff63dff4fc1b15ff1784a9749b7
I20260619 18:53:47.289108 23283 ts_tablet_manager.cc:1939] T 69518ff63dff4fc1b15ff1784a9749b7 P a60a1830734746bfa8ab432167bf9cef: Deleting consensus metadata
I20260619 18:53:47.291028 22675 catalog_manager.cc:4841] TS a60a1830734746bfa8ab432167bf9cef (127.15.91.3:38517): tablet 69518ff63dff4fc1b15ff1784a9749b7 (table test-table [id=de9af4cf5a39460d8c22804a2fa86361]) successfully deleted
I20260619 18:53:47.412825 23281 ts_tablet_manager.cc:1918] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:47.413023 23281 log.cc:1196] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/69518ff63dff4fc1b15ff1784a9749b7
I20260619 18:53:47.413525 23281 ts_tablet_manager.cc:1939] T 69518ff63dff4fc1b15ff1784a9749b7 P d1f612cc551a4147868957b40b15baf8: Deleting consensus metadata
I20260619 18:53:47.413748 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 22663
I20260619 18:53:47.432485 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 22882
I20260619 18:53:47.433517 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.433035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.0.0.1:41360)
0619 18:53:47.433223 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.433333 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 18:53:47.433971 22675 catalog_manager.cc:4584] TS d1f612cc551a4147868957b40b15baf8 (127.15.91.1:35739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69518ff63dff4fc1b15ff1784a9749b7: Network error: Client connection negotiation failed: client connection to 127.15.91.1:35739: connect: Connection refused (error 111)
I20260619 18:53:47.450337 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.433498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.15.91.2:39293)
0619 18:53:47.433733 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:47.433738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:47.433757 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:53:47.433952 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:47.433959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:47.434085 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:47.434205 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:47.436143 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.436994 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.439141 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.439743 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.439863 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:47.443416 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:47.443432 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:47.443449 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:47.443491 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:47.446897 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:47.447995 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:47.448003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:47.448006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:47.448184 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:47.448636 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:47.448640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:47.448642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:47.449032 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:47.449266 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:47.449655 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 18:53:47.449931 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 18:53:47.451200 23013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:47.486521 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.485931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.0.0.1:41376)
0619 18:53:47.486173 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.486342 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 18:53:47.487764 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.487276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.0.0.1:41380)
0619 18:53:47.487498 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.487594 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:53:47.495534 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.495023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:49970)
0619 18:53:47.495216 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.495327 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 18:53:47.496701 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.496270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:34269 (local address 127.0.0.1:49984)
0619 18:53:47.496458 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.496552 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 18:53:47.557823 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.557296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.0.0.1:41392)
0619 18:53:47.557542 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.557634 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 18:53:47.558974 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.558573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.0.0.1:41406)
0619 18:53:47.558765 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.558852 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:53:47.575204 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 22882
I20260619 18:53:47.596608 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 23031
I20260619 18:53:47.613066 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.597507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.15.91.3:38517)
0619 18:53:47.597685 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:47.597689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:47.597735 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:53:47.597792 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:47.597796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:47.597881 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:47.597963 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:47.599574 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.600312 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.601650 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:47.602098 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:47.602223 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:47.606046 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:47.606064 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:47.606081 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:47.606126 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:47.609838 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:47.610907 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:47.610916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:47.610918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:47.611097 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:47.611519 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:47.611523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:47.611525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:47.611914 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:47.612153 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:47.612604 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 18:53:47.612805 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:53:47.613852 23155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:47.666947 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.666291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.0.0.1:41422)
0619 18:53:47.666558 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.666654 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 18:53:47.668079 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.667719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.0.0.1:41430)
0619 18:53:47.667857 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.667948 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:53:47.739179 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 23031
I20260619 18:53:47.762351 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 22363
I20260619 18:53:47.763676 22531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:47.846874 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.846275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.0.0.1:41432)
0619 18:53:47.846538 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.846648 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 18:53:47.848023 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:47.847617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:35739 (local address 127.0.0.1:41440)
0619 18:53:47.847786 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:47.847879 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 18:53:47.885524 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 22363
I20260619 18:53:47.907959 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 22462
I20260619 18:53:47.909133 22721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:48.032886 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 22462
2026-06-19T18:53:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10408 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:53:48 dist-test-slave-dm3f krb5kdc[23292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:53:48 dist-test-slave-dm3f krb5kdc[23292](info): set up 2 sockets
Jun 19 18:53:48 dist-test-slave-dm3f krb5kdc[23292](info): commencing operation
krb5kdc: starting...
W20260619 18:53:49.136075 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:53:49 dist-test-slave-dm3f krb5kdc[23292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.62.keytab.
Entry for principal oryx/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.62.keytab.
I20260619 18:53:49.373842 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:43367
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.62.keytab
--principal=oryx/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:43367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:49.732342 23305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:49.732748 23305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:49.732923 23305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:49.741240 23305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:49.741428 23305 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:53:49.741531 23305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:49.741655 23305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:49.741751 23305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:49.741847 23305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:49.754288 23305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:43367
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.62.keytab
--principal=oryx/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:43367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:49.755882 23305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:49.757406 23305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:49.771059 23313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:49.771082 23311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:49.771049 23310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:49.772461 23305 server_base.cc:1034] running on GCE node
I20260619 18:53:49.773378 23305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:49.775115 23305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:49.776432 23305 hybrid_clock.cc:648] HybridClock initialized: now 1781895229776347 us; error 53 us; skew 500 ppm
I20260619 18:53:49.776889 23305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:49 dist-test-slave-dm3f krb5kdc[23292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895229, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:49.780871 23305 init.cc:377] Logged in from keytab as oryx/127.15.91.62@KRBTEST.COM (short username oryx)
I20260619 18:53:49.782883 23305 webserver.cc:460] Webserver started at http://127.15.91.62:40523/ using document root <none> and password file <none>
I20260619 18:53:49.783756 23305 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:49.783905 23305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:49.784307 23305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:49.787882 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "dc44ddc229764208a446506651b16744"
format_stamp: "Formatted at 2026-06-19 18:53:49 on dist-test-slave-dm3f"
server_key: "f4b43463f99ea396bc101ab65235f8ac"
server_key_iv: "71973cfcfdcd77c91698b3e2dfa3482a"
server_key_version: "encryptionkey@0"
I20260619 18:53:49.788928 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "dc44ddc229764208a446506651b16744"
format_stamp: "Formatted at 2026-06-19 18:53:49 on dist-test-slave-dm3f"
server_key: "f4b43463f99ea396bc101ab65235f8ac"
server_key_iv: "71973cfcfdcd77c91698b3e2dfa3482a"
server_key_version: "encryptionkey@0"
I20260619 18:53:49.795369 23305 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:53:49.799788 23320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:49.801573 23305 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:53:49.801785 23305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "dc44ddc229764208a446506651b16744"
format_stamp: "Formatted at 2026-06-19 18:53:49 on dist-test-slave-dm3f"
server_key: "f4b43463f99ea396bc101ab65235f8ac"
server_key_iv: "71973cfcfdcd77c91698b3e2dfa3482a"
server_key_version: "encryptionkey@0"
I20260619 18:53:49.802009 23305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:49.821297 23305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:49.825765 23305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:49.826017 23305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:49.850615 23305 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:43367
I20260619 18:53:49.850637 23381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:43367 every 8 connection(s)
I20260619 18:53:49.852317 23305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:49.856940 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:49.861238 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 23305
I20260619 18:53:49.861531 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:49.862610 15724 external_mini_cluster.cc:1428] Setting key de9e1e49d3b489bc963a309c781fd286
I20260619 18:53:49.870149 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744: Bootstrap starting.
I20260619 18:53:49.874524 23382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:49.875952 23382 log.cc:824] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:53:49 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:49.880195 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744: No bootstrap required, opened a new log
I20260619 18:53:49.891469 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:49.865100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.15.91.62:43367)
0619 18:53:49.866039 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:49.866057 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:49.866105 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:53:49.868195 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:49.868229 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:49.868761 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:49.869183 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:49.871387 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:49.872851 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:49.874867 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:49.875735 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:49.875941 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:49.880951 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:49.880964 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:49.880996 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:49.881065 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:49.885619 (+ 4554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:49.886628 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:49.886643 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:49.886659 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:49.886969 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:49.887422 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:49.887475 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:49.887478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:49.888445 (+ 967us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:49.888836 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:49.889209 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 18:53:49.889772 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":429,"threads_started":1}
I20260619 18:53:49.892153 23382 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc44ddc229764208a446506651b16744" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43367 } }
I20260619 18:53:49.892674 23382 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:49.892807 23382 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc44ddc229764208a446506651b16744, State: Initialized, Role: FOLLOWER
I20260619 18:53:49.893462 23382 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc44ddc229764208a446506651b16744" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43367 } }
I20260619 18:53:49.893792 23382 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:49.893961 23382 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:49.894165 23382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:49.896930 23382 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc44ddc229764208a446506651b16744" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43367 } }
I20260619 18:53:49.897574 23382 leader_election.cc:302] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc44ddc229764208a446506651b16744; no voters:
I20260619 18:53:49.898308 23382 leader_election.cc:288] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:49.898474 23387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:49.899446 23387 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [term 1 LEADER]: Becoming Leader. State: Replica: dc44ddc229764208a446506651b16744, State: Running, Role: LEADER
I20260619 18:53:49.900065 23387 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc44ddc229764208a446506651b16744" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43367 } }
I20260619 18:53:49.900796 23382 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:49.904718 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc44ddc229764208a446506651b16744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc44ddc229764208a446506651b16744" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43367 } } }
I20260619 18:53:49.905238 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:49.904769 23389 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc44ddc229764208a446506651b16744. Latest consensus state: current_term: 1 leader_uuid: "dc44ddc229764208a446506651b16744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc44ddc229764208a446506651b16744" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43367 } } }
I20260619 18:53:49.905807 23389 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744 [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:49.915644 23393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:49.922569 23393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:49.935459 23393 catalog_manager.cc:1347] Generated new cluster ID: 2db358c198e04e3a866a2a51d1c4eb01
I20260619 18:53:49.935595 23393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:49.967728 23393 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:49.969272 23393 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:49.977849 23393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744: Generated new TSK 0
I20260619 18:53:49.978963 23393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab.
Entry for principal oryx/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab.
I20260619 18:53:50.046536 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43367
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab
--principal=oryx/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:50.464911 23408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:50.465237 23408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:50.465415 23408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:50.473340 23408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:50.473495 23408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:50.473663 23408 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:53:50.473800 23408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:53:50.485106 23408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab
--principal=oryx/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.91.62:43367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:50.486588 23408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:50.488183 23408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:50.504089 23413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:50.504047 23416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:50.504024 23414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:50.505637 23408 server_base.cc:1034] running on GCE node
I20260619 18:53:50.506604 23408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:50.508071 23408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:50.509374 23408 hybrid_clock.cc:648] HybridClock initialized: now 1781895230509276 us; error 74 us; skew 500 ppm
I20260619 18:53:50.509783 23408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:50 dist-test-slave-dm3f krb5kdc[23292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895230, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:50.518234 23408 init.cc:377] Logged in from keytab as oryx/127.15.91.1@KRBTEST.COM (short username oryx)
I20260619 18:53:50.520641 23408 webserver.cc:460] Webserver started at http://127.15.91.1:33869/ using document root <none> and password file <none>
I20260619 18:53:50.521642 23408 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:50.521768 23408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:50.522172 23408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:50.525744 23408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "e7e073e2c830491a86ae0ee0388a56c8"
format_stamp: "Formatted at 2026-06-19 18:53:50 on dist-test-slave-dm3f"
server_key: "201953680c4b7972eac6fb8ac1729506"
server_key_iv: "ffb85eeff7ae1cea3ae635935dd4d701"
server_key_version: "encryptionkey@0"
I20260619 18:53:50.526870 23408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "e7e073e2c830491a86ae0ee0388a56c8"
format_stamp: "Formatted at 2026-06-19 18:53:50 on dist-test-slave-dm3f"
server_key: "201953680c4b7972eac6fb8ac1729506"
server_key_iv: "ffb85eeff7ae1cea3ae635935dd4d701"
server_key_version: "encryptionkey@0"
I20260619 18:53:50.533883 23408 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 18:53:50.538563 23423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:50.540452 23408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260619 18:53:50.540701 23408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "e7e073e2c830491a86ae0ee0388a56c8"
format_stamp: "Formatted at 2026-06-19 18:53:50 on dist-test-slave-dm3f"
server_key: "201953680c4b7972eac6fb8ac1729506"
server_key_iv: "ffb85eeff7ae1cea3ae635935dd4d701"
server_key_version: "encryptionkey@0"
I20260619 18:53:50.541000 23408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:50.556774 23408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:50.562000 23408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:50.562345 23408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:50.565548 23408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:50.565690 23408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:50.565913 23408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:50.566027 23408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 18:53:50 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895230, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:50.594872 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:50.566167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:59279 (local address 127.15.91.62:43367)
0619 18:53:50.567225 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:50.567233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:50.568377 (+ 1144us) server_negotiation.cc:407] Connection header received
0619 18:53:50.570751 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:50.570761 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:50.570896 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:50.571053 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:50.574009 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:50.575432 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:50.578288 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:50.578987 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:50.579181 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:50.586147 (+ 6966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:50.586164 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:50.586167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:50.586209 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:50.590117 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:50.591330 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:50.591338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:50.591342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:50.591541 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:50.592318 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:50.592323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:50.592327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:50.592807 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:50.592980 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:50.594141 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0619 18:53:50.594434 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":694,"threads_started":1}
I20260619 18:53:50.597198 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:50.567059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43367 (local address 127.15.91.1:59279)
0619 18:53:50.568122 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:50.568190 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:50.570325 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:50.571409 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:50.571422 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:50.572433 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:50.573658 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:50.573709 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:50.575726 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:50.575739 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:50.578032 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:50.578055 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:50.579599 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:50.579605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:50.580078 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:50.581051 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:50.581100 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:50.585718 (+ 4618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:50.590352 (+ 4634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:50.590372 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:50.590392 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:50.591068 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:50.591706 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:50.591711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:50.591713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:50.592144 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:50.592987 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:50.592999 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:50.593661 (+ 662us) client_negotiation.cc:769] Sending connection context
0619 18:53:50.594012 (+ 351us) client_negotiation.cc:240] Negotiation successful
0619 18:53:50.594698 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":437,"threads_started":1}
I20260619 18:53:50.620129 23408 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:44671
I20260619 18:53:50.620277 23543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:44671 every 8 connection(s)
I20260619 18:53:50.622119 23408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:53:50.624814 23544 heartbeater.cc:347] Connected to a master server at 127.15.91.62:43367
I20260619 18:53:50.625128 23544 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:50.625931 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:50.606218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:50767 (local address 127.15.91.62:43367)
0619 18:53:50.606471 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:50.606481 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:50.607828 (+ 1347us) server_negotiation.cc:407] Connection header received
0619 18:53:50.608106 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:50.608114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:50.608216 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:50.608316 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:50.609948 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:50.611111 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:50.613525 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:50.614325 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:50.614506 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:50.617743 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:50.617751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:50.617755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:50.617802 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:50.622322 (+ 4520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:50.623536 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:50.623545 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:50.623548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:50.623745 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:50.624226 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:50.624231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:50.624236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:50.624567 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:50.624775 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:50.625272 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 18:53:50.625526 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:53:50.625974 23544 heartbeater.cc:511] Master 127.15.91.62:43367 requested a full tablet report, sending...
I20260619 18:53:50.626082 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:50.606169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43367 (local address 127.15.91.1:50767)
0619 18:53:50.607621 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:50.607658 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:50.607936 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:50.608419 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:50.608427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:50.609218 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:50.609690 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:50.609707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:50.611308 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:50.611317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:50.613285 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:50.613307 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:50.614577 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:50.614585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:50.614865 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:50.615661 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:50.615694 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:50.617384 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:50.622542 (+ 5158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:50.622547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:50.622550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:50.623305 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:50.623932 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:50.623943 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:50.623945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:50.624083 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:50.624713 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:50.624719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:50.624967 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 18:53:50.625155 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 18:53:50.625558 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"thread_start_us":888,"threads_started":1}
I20260619 18:53:50.628077 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 23408
I20260619 18:53:50.628298 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:53:50.628767 23336 ts_manager.cc:194] Registered new tserver with Master: e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1:44671)
I20260619 18:53:50.629303 15724 external_mini_cluster.cc:1428] Setting key 0a33794226615358c0ecd1a0eb58bf2c
I20260619 18:53:50.630831 23336 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:59279
WARNING: no policy specified for oryx/127.15.91.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab.
Entry for principal oryx/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab.
I20260619 18:53:50.684134 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43367
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab
--principal=oryx/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:51.058079 23548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:51.058414 23548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:51.058531 23548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:51.066449 23548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:51.066602 23548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:51.066689 23548 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:53:51.066862 23548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:53:51.077513 23548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab
--principal=oryx/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.91.62:43367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:51.078991 23548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:51.080534 23548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:51.096140 23553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:51.096140 23556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:51.096180 23554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:51.097307 23548 server_base.cc:1034] running on GCE node
I20260619 18:53:51.098196 23548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:51.099542 23548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:51.100847 23548 hybrid_clock.cc:648] HybridClock initialized: now 1781895231100723 us; error 105 us; skew 500 ppm
I20260619 18:53:51.101325 23548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:51 dist-test-slave-dm3f krb5kdc[23292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895231, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:51.106880 23548 init.cc:377] Logged in from keytab as oryx/127.15.91.2@KRBTEST.COM (short username oryx)
I20260619 18:53:51.108703 23548 webserver.cc:460] Webserver started at http://127.15.91.2:43155/ using document root <none> and password file <none>
I20260619 18:53:51.109620 23548 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:51.109795 23548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:51.110183 23548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:51.113646 23548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "f3c1da409f4541dbaa0e7b5522886166"
format_stamp: "Formatted at 2026-06-19 18:53:51 on dist-test-slave-dm3f"
server_key: "a391530da49729ef9d2000145a8708d5"
server_key_iv: "c0b356c46ce304e67628022923c33deb"
server_key_version: "encryptionkey@0"
I20260619 18:53:51.114851 23548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "f3c1da409f4541dbaa0e7b5522886166"
format_stamp: "Formatted at 2026-06-19 18:53:51 on dist-test-slave-dm3f"
server_key: "a391530da49729ef9d2000145a8708d5"
server_key_iv: "c0b356c46ce304e67628022923c33deb"
server_key_version: "encryptionkey@0"
I20260619 18:53:51.121336 23548 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 18:53:51.126487 23563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:51.128439 23548 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:53:51.128721 23548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "f3c1da409f4541dbaa0e7b5522886166"
format_stamp: "Formatted at 2026-06-19 18:53:51 on dist-test-slave-dm3f"
server_key: "a391530da49729ef9d2000145a8708d5"
server_key_iv: "c0b356c46ce304e67628022923c33deb"
server_key_version: "encryptionkey@0"
I20260619 18:53:51.129012 23548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:51.153152 23548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:51.158206 23548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:51.158555 23548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:51.161863 23548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:51.162016 23548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:51.162169 23548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:51.162243 23548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:53:51 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895231, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:51.193766 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.162704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:47563 (local address 127.15.91.62:43367)
0619 18:53:51.163344 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:51.163350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:51.164802 (+ 1452us) server_negotiation.cc:407] Connection header received
0619 18:53:51.167277 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:51.167283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:51.167396 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:51.167551 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:51.170255 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.171805 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.175230 (+ 3425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.176085 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.176286 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.182278 (+ 5992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:51.182289 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:51.182295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:51.182361 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:51.187374 (+ 5013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:51.189308 (+ 1934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:51.189315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:51.189317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:51.189609 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:51.191294 (+ 1685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:51.191306 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:51.191310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:51.191802 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:51.192109 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:51.193048 (+ 939us) server_negotiation.cc:299] Negotiation successful
0619 18:53:51.193352 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":373,"threads_started":1}
I20260619 18:53:51.195057 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.163596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43367 (local address 127.15.91.2:47563)
0619 18:53:51.164587 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:51.164638 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:51.166887 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:51.167861 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:51.167872 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:51.168774 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:51.169911 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.169963 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.172153 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.172161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.174932 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.174973 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.176496 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.176506 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.176788 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.177871 (+ 1083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:51.177929 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:51.181877 (+ 3948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:51.187757 (+ 5880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:51.187862 (+ 105us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:51.187911 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:51.189009 (+ 1098us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:51.189874 (+ 865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:51.189886 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:51.189894 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:51.190489 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:51.192042 (+ 1553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:51.192053 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:51.192577 (+ 524us) client_negotiation.cc:769] Sending connection context
0619 18:53:51.192907 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 18:53:51.193552 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":414,"threads_started":1}
I20260619 18:53:51.216199 23548 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:44971
I20260619 18:53:51.216327 23683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:44971 every 8 connection(s)
I20260619 18:53:51.218323 23548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:53:51.221906 23684 heartbeater.cc:347] Connected to a master server at 127.15.91.62:43367
I20260619 18:53:51.222285 23684 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:51.223238 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 23548
I20260619 18:53:51.223459 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:53:51.223536 23684 heartbeater.cc:511] Master 127.15.91.62:43367 requested a full tablet report, sending...
I20260619 18:53:51.224117 15724 external_mini_cluster.cc:1428] Setting key 89bb79278ebd03c5b70a2a3e70ad22ff
I20260619 18:53:51.225709 23336 ts_manager.cc:194] Registered new tserver with Master: f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2:44971)
I20260619 18:53:51.226045 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.206392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:45087 (local address 127.15.91.62:43367)
0619 18:53:51.206628 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:51.206637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:51.207419 (+ 782us) server_negotiation.cc:407] Connection header received
0619 18:53:51.207664 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:51.207675 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:51.207784 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:51.207946 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:51.209805 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.211208 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.213976 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.214813 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.215020 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.218079 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:51.218087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:51.218092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:51.218143 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:51.222099 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:51.223255 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:51.223265 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:51.223272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:51.223498 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:51.224071 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:51.224078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:51.224084 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:51.224591 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:51.224824 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:51.225399 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 18:53:51.225669 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:53:51.226161 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.206443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43367 (local address 127.15.91.2:45087)
0619 18:53:51.207245 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:51.207271 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:51.207507 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:51.208004 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:51.208014 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:51.208939 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:51.209521 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.209545 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.211471 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.211482 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.213715 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.213749 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.215077 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.215085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.215338 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.216280 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:51.216331 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:51.217283 (+ 952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:51.222271 (+ 4988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:51.222280 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:51.222285 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:51.223058 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:51.223680 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:51.223685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:51.223689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:51.223874 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:51.224783 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:51.224789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:51.225038 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 18:53:51.225234 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 18:53:51.225652 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":464,"threads_started":1}
I20260619 18:53:51.226941 23336 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:47563
WARNING: no policy specified for oryx/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab.
Entry for principal oryx/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab.
I20260619 18:53:51.284065 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43367
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab
--principal=oryx/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:53:51.635030 23544 heartbeater.cc:503] Master 127.15.91.62:43367 was elected leader, sending a full tablet report...
W20260619 18:53:51.661027 23688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:51.661458 23688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:51.661588 23688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:51.670774 23688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:51.670933 23688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:51.671032 23688 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:53:51.671123 23688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:53:51.681841 23688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab
--principal=oryx/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.91.62:43367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:51.683240 23688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:51.684957 23688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:51.701143 23694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:51.701538 23696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:51.701135 23693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:51.703848 23688 server_base.cc:1034] running on GCE node
I20260619 18:53:51.704708 23688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:51.705986 23688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:51.707329 23688 hybrid_clock.cc:648] HybridClock initialized: now 1781895231707224 us; error 56 us; skew 500 ppm
I20260619 18:53:51.707758 23688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:51 dist-test-slave-dm3f krb5kdc[23292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895231, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:51.711771 23688 init.cc:377] Logged in from keytab as oryx/127.15.91.3@KRBTEST.COM (short username oryx)
I20260619 18:53:51.713670 23688 webserver.cc:460] Webserver started at http://127.15.91.3:37677/ using document root <none> and password file <none>
I20260619 18:53:51.714622 23688 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:51.714848 23688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:51.715312 23688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:51.718628 23688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "4594e77ee70b46cfa03352fb496a3cc6"
format_stamp: "Formatted at 2026-06-19 18:53:51 on dist-test-slave-dm3f"
server_key: "ec07dac3618216e552d1d73c7b2b044b"
server_key_iv: "d2b45533c7bf879c9d609502e55b9384"
server_key_version: "encryptionkey@0"
I20260619 18:53:51.719739 23688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "4594e77ee70b46cfa03352fb496a3cc6"
format_stamp: "Formatted at 2026-06-19 18:53:51 on dist-test-slave-dm3f"
server_key: "ec07dac3618216e552d1d73c7b2b044b"
server_key_iv: "d2b45533c7bf879c9d609502e55b9384"
server_key_version: "encryptionkey@0"
I20260619 18:53:51.726318 23688 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:53:51.731372 23703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:51.733762 23688 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:53:51.733971 23688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "4594e77ee70b46cfa03352fb496a3cc6"
format_stamp: "Formatted at 2026-06-19 18:53:51 on dist-test-slave-dm3f"
server_key: "ec07dac3618216e552d1d73c7b2b044b"
server_key_iv: "d2b45533c7bf879c9d609502e55b9384"
server_key_version: "encryptionkey@0"
I20260619 18:53:51.734320 23688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:51.752516 23688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:51.759229 23688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:51.759579 23688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:51.763833 23688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:51.764117 23688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 18:53:51.764338 23688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:51.764480 23688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:53:51 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895231, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:51.794211 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.764146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:39081 (local address 127.15.91.62:43367)
0619 18:53:51.764850 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:51.764859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:51.766787 (+ 1928us) server_negotiation.cc:407] Connection header received
0619 18:53:51.770307 (+ 3520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:51.770316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:51.770459 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:51.770613 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:51.773671 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.774820 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.777782 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.778245 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.778409 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.784582 (+ 6173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:51.784588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:51.784592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:51.784633 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:51.788664 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:51.790180 (+ 1516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:51.790186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:51.790188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:51.790376 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:51.791172 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:51.791178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:51.791181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:51.791546 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:51.791781 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:51.793365 (+ 1584us) server_negotiation.cc:299] Negotiation successful
0619 18:53:51.793728 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":283,"threads_started":1}
I20260619 18:53:51.796599 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.764930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43367 (local address 127.15.91.3:39081)
0619 18:53:51.766311 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:51.766401 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:51.769771 (+ 3370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:51.771062 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:51.771081 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:51.772149 (+ 1068us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:51.773375 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.773424 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.775057 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.775061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.777537 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.777562 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.778573 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.778587 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.778986 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.780108 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:51.780155 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:51.784230 (+ 4075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:51.788977 (+ 4747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:51.788996 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:51.789018 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:51.789963 (+ 945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:51.790522 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:51.790530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:51.790536 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:51.791007 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:51.791785 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:51.791803 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:51.792574 (+ 771us) client_negotiation.cc:769] Sending connection context
0619 18:53:51.793143 (+ 569us) client_negotiation.cc:240] Negotiation successful
0619 18:53:51.794048 (+ 905us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":511,"threads_started":1}
I20260619 18:53:51.816033 23688 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:44427
I20260619 18:53:51.816045 23823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:44427 every 8 connection(s)
I20260619 18:53:51.817615 23688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:53:51.820832 23824 heartbeater.cc:347] Connected to a master server at 127.15.91.62:43367
I20260619 18:53:51.821192 23824 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:51.822173 23824 heartbeater.cc:511] Master 127.15.91.62:43367 requested a full tablet report, sending...
I20260619 18:53:51.822458 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 23688
I20260619 18:53:51.822944 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:53:51.823573 15724 external_mini_cluster.cc:1428] Setting key c62df0e94ba83ccf78fbfd1651012e61
I20260619 18:53:51.824011 23337 ts_manager.cc:194] Registered new tserver with Master: 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427)
I20260619 18:53:51.825215 23337 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:39081
I20260619 18:53:51.827428 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:53:51.834769 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.805063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:53183 (local address 127.15.91.62:43367)
0619 18:53:51.805342 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:51.805349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:51.820043 (+ 14694us) server_negotiation.cc:407] Connection header received
0619 18:53:51.820225 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:51.820231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:51.820357 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:51.820502 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:51.822039 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.823562 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.825552 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.826302 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.826451 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.828696 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:51.828702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:51.828705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:51.828741 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:51.831713 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:51.832546 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:51.832553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:51.832559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:51.832708 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:51.833125 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:51.833131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:51.833136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:51.833496 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:51.833666 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:51.834118 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 18:53:51.834376 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 18:53:51.834879 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.804959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43367 (local address 127.15.91.3:53183)
0619 18:53:51.819893 (+ 14934us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:51.819918 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:51.820077 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:51.820540 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:51.820546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:51.821274 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:51.821697 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.821717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.823784 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.823791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.825354 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.825378 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.826538 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.826546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.826782 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.827635 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:51.827655 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:51.828430 (+ 775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:51.831939 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:51.831946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:51.831950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:51.832396 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:51.832837 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:51.832850 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:51.832854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:51.833002 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:51.833622 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:51.833627 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:51.833829 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 18:53:51.833998 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 18:53:51.834392 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14750,"thread_start_us":15187,"threads_started":1}
Jun 19 18:53:51 dist-test-slave-dm3f krb5kdc[23292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895231, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:53:51 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:53:51 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:53:51.890244 23711 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:53:51.877779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.15.91.62:43367)
0619 18:53:51.878042 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:51.878047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:51.878362 (+ 315us) server_negotiation.cc:407] Connection header received
0619 18:53:51.878602 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:51.878608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:51.878810 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:51.878927 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:51.880536 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.881294 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.882950 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.883409 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.883553 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.889985 (+ 6432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42738: BlockingRecv error: recv got EOF from 127.0.0.1:42738 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Jun 19 18:53:51 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895231, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:51.910688 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.894319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.15.91.62:43367)
0619 18:53:51.894533 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:51.894538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:51.894880 (+ 342us) server_negotiation.cc:407] Connection header received
0619 18:53:51.895047 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:51.895050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:51.895118 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:51.895213 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:51.896517 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.897252 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.898844 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.899368 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.899513 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.904146 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:51.904152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:51.904156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:51.904203 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:51.907276 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:51.908221 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:51.908225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:51.908228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:51.908397 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:51.908883 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:51.908887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:51.908890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:51.909196 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:51.909479 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:51.910206 (+ 727us) server_negotiation.cc:299] Negotiation successful
0619 18:53:51.910430 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:53:51.915544 23337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:53:51.923007 23337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:51.958878 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.945055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:44427 (local address 127.0.0.1:55542)
0619 18:53:51.945772 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:51.945798 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:51.945932 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:51.948157 (+ 2225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:51.948171 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:51.948227 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:51.948847 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.948867 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.952020 (+ 3153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.952031 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.955639 (+ 3608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.955666 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.957726 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.957733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.958014 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.958087 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 18:53:51.958234 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 18:53:51.958451 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":2081,"threads_started":1}
I20260619 18:53:51.958889 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.946406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.15.91.3:44427)
0619 18:53:51.947232 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:51.947240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:51.947277 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:53:51.947402 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:51.947411 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:51.947775 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:51.948108 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:51.949191 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.951680 (+ 2489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.955909 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.957447 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.957640 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.957903 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:51.958308 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 18:53:51.958495 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"spinlock_wait_cycles":28032,"thread_start_us":551,"threads_started":1}
I20260619 18:53:51.964555 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.946074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.15.91.2:44971)
0619 18:53:51.946860 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:51.946874 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:51.956052 (+ 9178us) server_negotiation.cc:407] Connection header received
0619 18:53:51.956440 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:51.956452 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:51.956714 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:51.956913 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:51.957866 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.959310 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.961975 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.963199 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.963298 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.963455 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:51.964045 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 18:53:51.964220 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":693,"threads_started":1}
I20260619 18:53:51.964576 23845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.945780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:44971 (local address 127.0.0.1:58350)
0619 18:53:51.955447 (+ 9667us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:51.955483 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:51.956188 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:51.957211 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:51.957215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:51.957241 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:51.957632 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.957646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.959514 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.959518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.961752 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.961766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.963661 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.963666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.963802 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.963827 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:53:51.963963 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 18:53:51.964140 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9486,"thread_start_us":9291,"threads_started":1}
I20260619 18:53:51.964641 23842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.944450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:44671 (local address 127.0.0.1:34684)
0619 18:53:51.945286 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:51.945336 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:51.945585 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:51.956573 (+ 10988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:51.956579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:51.956608 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:51.957053 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.957068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.959404 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.959412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.961554 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:51.961577 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.963730 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.963738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:51.963923 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.963961 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 18:53:51.964044 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:53:51.964198 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":29,"thread_start_us":392,"threads_started":1}
I20260619 18:53:51.964159 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 65032b019f274447a37af4f4ddcce6b3 (DEFAULT_TABLE table=test-table [id=90eaea3224344aa98d8b9eb082048291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:51.965853 23843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:51.944615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.15.91.1:44671)
0619 18:53:51.955760 (+ 11145us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:51.955771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:51.955808 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:53:51.955932 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:51.955942 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:51.956267 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:51.957271 (+ 1004us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:51.957513 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.959220 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.961838 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:51.963404 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:51.964933 (+ 1529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:51.965199 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:51.965293 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 18:53:51.965509 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10964,"thread_start_us":10810,"threads_started":1}
I20260619 18:53:51.966154 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65032b019f274447a37af4f4ddcce6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:51.969342 23474 tablet_service.cc:1467] Processing CreateTablet for tablet 65032b019f274447a37af4f4ddcce6b3 (DEFAULT_TABLE table=test-table [id=90eaea3224344aa98d8b9eb082048291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:51.969334 23614 tablet_service.cc:1467] Processing CreateTablet for tablet 65032b019f274447a37af4f4ddcce6b3 (DEFAULT_TABLE table=test-table [id=90eaea3224344aa98d8b9eb082048291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:51.970904 23474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65032b019f274447a37af4f4ddcce6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:51.970919 23614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65032b019f274447a37af4f4ddcce6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:51.984484 23848 tablet_bootstrap.cc:492] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8: Bootstrap starting.
I20260619 18:53:51.987068 23849 tablet_bootstrap.cc:492] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166: Bootstrap starting.
I20260619 18:53:51.990708 23848 tablet_bootstrap.cc:654] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:51.990751 23849 tablet_bootstrap.cc:654] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:51.992213 23850 tablet_bootstrap.cc:492] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: Bootstrap starting.
I20260619 18:53:51.992369 23849 log.cc:824] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:51.992369 23848 log.cc:824] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:51.996518 23850 tablet_bootstrap.cc:654] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:51.996833 23849 tablet_bootstrap.cc:492] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166: No bootstrap required, opened a new log
I20260619 18:53:51.997190 23849 ts_tablet_manager.cc:1397] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 18:53:51.998071 23850 log.cc:824] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:51.998342 23848 tablet_bootstrap.cc:492] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8: No bootstrap required, opened a new log
I20260619 18:53:51.998677 23848 ts_tablet_manager.cc:1397] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260619 18:53:52.002918 23850 tablet_bootstrap.cc:492] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: No bootstrap required, opened a new log
I20260619 18:53:52.003378 23850 ts_tablet_manager.cc:1397] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20260619 18:53:52.006582 23849 raft_consensus.cc:348] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } }
I20260619 18:53:52.007155 23849 raft_consensus.cc:374] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.007316 23849 raft_consensus.cc:729] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c1da409f4541dbaa0e7b5522886166, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.008106 23849 consensus_queue.cc:260] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } }
I20260619 18:53:52.009405 23684 heartbeater.cc:503] Master 127.15.91.62:43367 was elected leader, sending a full tablet report...
I20260619 18:53:52.012131 23849 ts_tablet_manager.cc:1428] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260619 18:53:52.012616 23848 raft_consensus.cc:348] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } }
I20260619 18:53:52.013159 23848 raft_consensus.cc:374] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.013265 23848 raft_consensus.cc:729] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e073e2c830491a86ae0ee0388a56c8, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.014038 23848 consensus_queue.cc:260] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } }
I20260619 18:53:52.016285 23848 ts_tablet_manager.cc:1428] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260619 18:53:52.016270 23850 raft_consensus.cc:348] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } }
I20260619 18:53:52.017019 23850 raft_consensus.cc:374] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.017179 23850 raft_consensus.cc:729] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4594e77ee70b46cfa03352fb496a3cc6, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.018002 23850 consensus_queue.cc:260] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } }
I20260619 18:53:52.019444 23824 heartbeater.cc:503] Master 127.15.91.62:43367 was elected leader, sending a full tablet report...
I20260619 18:53:52.020118 23850 ts_tablet_manager.cc:1428] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260619 18:53:52.020557 23854 raft_consensus.cc:482] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:52.020880 23854 raft_consensus.cc:504] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } }
I20260619 18:53:52.023264 23854 leader_election.cc:288] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1:44671), 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427)
I20260619 18:53:52.035043 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.023945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:44671 (local address 127.15.91.2:36115)
0619 18:53:52.027132 (+ 3187us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:52.027180 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:52.027406 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:52.027949 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:52.027955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:52.027984 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:52.028510 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.028525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.029662 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.029672 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.032543 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.032575 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.034121 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.034128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.034277 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.034309 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:53:52.034505 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 18:53:52.034644 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3053,"mutex_wait_us":163,"thread_start_us":2535,"threads_started":1}
I20260619 18:53:52.035034 23843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.024896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:36115 (local address 127.15.91.1:44671)
0619 18:53:52.025094 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:52.025100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:52.027280 (+ 2180us) server_negotiation.cc:407] Connection header received
0619 18:53:52.027560 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:52.027566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:52.027659 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:52.027764 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:52.028703 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.029444 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.032873 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.033768 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.034028 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.034147 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:52.034560 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 18:53:52.034744 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:53:52.036414 23494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65032b019f274447a37af4f4ddcce6b3" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e073e2c830491a86ae0ee0388a56c8" is_pre_election: true
I20260619 18:53:52.036824 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.023960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:33299 (local address 127.15.91.3:44427)
0619 18:53:52.024582 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:52.024590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:52.027198 (+ 2608us) server_negotiation.cc:407] Connection header received
0619 18:53:52.027405 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:52.027412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:52.027503 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:52.027605 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:52.028337 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.029394 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.034033 (+ 4639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.035589 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.035752 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.035858 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:52.036341 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 18:53:52.036561 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464}
I20260619 18:53:52.036939 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.023532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:44427 (local address 127.15.91.2:33299)
0619 18:53:52.027053 (+ 3521us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:52.027078 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:52.027250 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:52.027618 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:52.027622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:52.027647 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:52.028111 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.028123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.029593 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.029599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.032543 (+ 2944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.032575 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.035908 (+ 3333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.035915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.036123 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.036170 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 18:53:52.036297 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 18:53:52.036498 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3326,"mutex_wait_us":181,"thread_start_us":1881,"threads_started":1}
I20260619 18:53:52.037106 23494 raft_consensus.cc:2454] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 0.
I20260619 18:53:52.037907 23567 leader_election.cc:302] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7e073e2c830491a86ae0ee0388a56c8, f3c1da409f4541dbaa0e7b5522886166; no voters:
I20260619 18:53:52.038405 23854 raft_consensus.cc:2781] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:52.038583 23854 raft_consensus.cc:482] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:52.038700 23854 raft_consensus.cc:3037] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.038578 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65032b019f274447a37af4f4ddcce6b3" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4594e77ee70b46cfa03352fb496a3cc6" is_pre_election: true
I20260619 18:53:52.039281 23774 raft_consensus.cc:2454] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 0.
I20260619 18:53:52.041527 23854 raft_consensus.cc:504] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } }
I20260619 18:53:52.042249 23854 leader_election.cc:288] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 election: Requested vote from peers e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1:44671), 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427)
I20260619 18:53:52.043031 23494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65032b019f274447a37af4f4ddcce6b3" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e073e2c830491a86ae0ee0388a56c8"
I20260619 18:53:52.043040 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65032b019f274447a37af4f4ddcce6b3" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4594e77ee70b46cfa03352fb496a3cc6"
I20260619 18:53:52.043282 23494 raft_consensus.cc:3037] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.043308 23774 raft_consensus.cc:3037] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.046084 23494 raft_consensus.cc:2454] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 1.
I20260619 18:53:52.046324 23774 raft_consensus.cc:2454] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 1.
I20260619 18:53:52.047003 23564 leader_election.cc:302] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4594e77ee70b46cfa03352fb496a3cc6, f3c1da409f4541dbaa0e7b5522886166; no voters:
I20260619 18:53:52.047657 23854 raft_consensus.cc:2781] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:52.048489 23854 raft_consensus.cc:686] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 1 LEADER]: Becoming Leader. State: Replica: f3c1da409f4541dbaa0e7b5522886166, State: Running, Role: LEADER
I20260619 18:53:52.049289 23854 consensus_queue.cc:237] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } }
I20260619 18:53:52.056651 23336 catalog_manager.cc:5495] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2). New cstate: current_term: 1 leader_uuid: "f3c1da409f4541dbaa0e7b5522886166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } health_report { overall_health: HEALTHY } } }
W20260619 18:53:52.069829 23825 tablet.cc:2369] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:53:52.090216 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.083021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.15.91.2:44971)
0619 18:53:52.083309 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:52.083316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:52.083471 (+ 155us) server_negotiation.cc:407] Connection header received
0619 18:53:52.083763 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:52.083767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:52.083868 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:52.084064 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:52.084500 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.085607 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.087716 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.088182 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.088357 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.088876 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:52.089495 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:52.089668 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:52.089809 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 18:53:52.089907 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 18:53:52.101717 23494 raft_consensus.cc:1261] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Refusing update from remote peer f3c1da409f4541dbaa0e7b5522886166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:52.101784 23774 raft_consensus.cc:1261] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Refusing update from remote peer f3c1da409f4541dbaa0e7b5522886166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:52.103492 23859 consensus_queue.cc:1035] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:52.103948 23854 consensus_queue.cc:1035] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:52.136504 23336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:52.136884 23336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:52.141355 23336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744: Sending DeleteTablet for 3 replicas of tablet 65032b019f274447a37af4f4ddcce6b3
I20260619 18:53:52.142817 23474 tablet_service.cc:1514] Processing DeleteTablet for tablet 65032b019f274447a37af4f4ddcce6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:52 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:34684
I20260619 18:53:52.142773 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet 65032b019f274447a37af4f4ddcce6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:52 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:55542
I20260619 18:53:52.143913 23872 tablet_replica.cc:330] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: stopping tablet replica
I20260619 18:53:52.144044 23614 tablet_service.cc:1514] Processing DeleteTablet for tablet 65032b019f274447a37af4f4ddcce6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:52 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:58350
I20260619 18:53:52.144258 23336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:53:52.144906 23871 tablet_replica.cc:330] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8: stopping tablet replica
W20260619 18:53:52.145053 23336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:52.145627 23872 raft_consensus.cc:2229] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:52.146914 23872 raft_consensus.cc:2258] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:52.146245 23871 raft_consensus.cc:2229] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:52.148468 23871 raft_consensus.cc:2258] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:52.151541 23872 ts_tablet_manager.cc:1905] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:52.154738 23474 tablet_service.cc:1467] Processing CreateTablet for tablet a8f075cacd484ad58f6bd12f4d8f8cf2 (DEFAULT_TABLE table=test-table [id=a5d869f3a7554f90b9c414e5bdce3987]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:52.154740 23754 tablet_service.cc:1467] Processing CreateTablet for tablet a8f075cacd484ad58f6bd12f4d8f8cf2 (DEFAULT_TABLE table=test-table [id=a5d869f3a7554f90b9c414e5bdce3987]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:52.155040 23871 ts_tablet_manager.cc:1905] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:52.155294 23474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f075cacd484ad58f6bd12f4d8f8cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:52.155398 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f075cacd484ad58f6bd12f4d8f8cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:52.156102 23613 tablet_service.cc:1467] Processing CreateTablet for tablet a8f075cacd484ad58f6bd12f4d8f8cf2 (DEFAULT_TABLE table=test-table [id=a5d869f3a7554f90b9c414e5bdce3987]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:52.156772 23613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f075cacd484ad58f6bd12f4d8f8cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:52.162808 23848 tablet_bootstrap.cc:492] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8: Bootstrap starting.
I20260619 18:53:52.165248 23848 tablet_bootstrap.cc:654] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:52.165227 23873 tablet_replica.cc:330] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166: stopping tablet replica
I20260619 18:53:52.165359 23849 tablet_bootstrap.cc:492] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166: Bootstrap starting.
I20260619 18:53:52.165814 23873 raft_consensus.cc:2229] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:52.166771 23873 raft_consensus.cc:2258] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:52.167347 23849 tablet_bootstrap.cc:654] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:52.169396 23850 tablet_bootstrap.cc:492] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6: Bootstrap starting.
I20260619 18:53:52.171593 23850 tablet_bootstrap.cc:654] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:52.175906 23872 ts_tablet_manager.cc:1918] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:52.176156 23872 log.cc:1196] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/65032b019f274447a37af4f4ddcce6b3
I20260619 18:53:52.176797 23872 ts_tablet_manager.cc:1939] T 65032b019f274447a37af4f4ddcce6b3 P 4594e77ee70b46cfa03352fb496a3cc6: Deleting consensus metadata
I20260619 18:53:52.177243 23871 ts_tablet_manager.cc:1918] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:52.177440 23873 ts_tablet_manager.cc:1905] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:52.177678 23871 log.cc:1196] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/65032b019f274447a37af4f4ddcce6b3
I20260619 18:53:52.178371 23871 ts_tablet_manager.cc:1939] T 65032b019f274447a37af4f4ddcce6b3 P e7e073e2c830491a86ae0ee0388a56c8: Deleting consensus metadata
I20260619 18:53:52.179050 23321 catalog_manager.cc:4841] TS 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427): tablet 65032b019f274447a37af4f4ddcce6b3 (table test-table [id=90eaea3224344aa98d8b9eb082048291]) successfully deleted
I20260619 18:53:52.181041 23848 tablet_bootstrap.cc:492] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8: No bootstrap required, opened a new log
I20260619 18:53:52.181399 23848 ts_tablet_manager.cc:1397] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.008s
I20260619 18:53:52.182621 23848 raft_consensus.cc:348] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.183024 23848 raft_consensus.cc:374] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.183183 23848 raft_consensus.cc:729] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e073e2c830491a86ae0ee0388a56c8, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.183631 23848 consensus_queue.cc:260] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.184432 23850 tablet_bootstrap.cc:492] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6: No bootstrap required, opened a new log
I20260619 18:53:52.184612 23850 ts_tablet_manager.cc:1397] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260619 18:53:52.185504 23850 raft_consensus.cc:348] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.185771 23850 raft_consensus.cc:374] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.185880 23850 raft_consensus.cc:729] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4594e77ee70b46cfa03352fb496a3cc6, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.186214 23848 ts_tablet_manager.cc:1428] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260619 18:53:52.186235 23850 consensus_queue.cc:260] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.186950 23850 ts_tablet_manager.cc:1428] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 18:53:52.188162 23324 catalog_manager.cc:4841] TS e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1:44671): tablet 65032b019f274447a37af4f4ddcce6b3 (table test-table [id=90eaea3224344aa98d8b9eb082048291]) successfully deleted
I20260619 18:53:52.192401 23849 tablet_bootstrap.cc:492] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166: No bootstrap required, opened a new log
I20260619 18:53:52.192629 23849 ts_tablet_manager.cc:1397] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.012s
I20260619 18:53:52.193462 23873 ts_tablet_manager.cc:1918] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:52.193698 23873 log.cc:1196] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/65032b019f274447a37af4f4ddcce6b3
I20260619 18:53:52.193727 23849 raft_consensus.cc:348] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.194034 23849 raft_consensus.cc:374] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.194154 23849 raft_consensus.cc:729] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c1da409f4541dbaa0e7b5522886166, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.194408 23873 ts_tablet_manager.cc:1939] T 65032b019f274447a37af4f4ddcce6b3 P f3c1da409f4541dbaa0e7b5522886166: Deleting consensus metadata
I20260619 18:53:52.194504 23849 consensus_queue.cc:260] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.195271 23849 ts_tablet_manager.cc:1428] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 18:53:52.196424 23324 catalog_manager.cc:4841] TS f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2:44971): tablet 65032b019f274447a37af4f4ddcce6b3 (table test-table [id=90eaea3224344aa98d8b9eb082048291]) successfully deleted
W20260619 18:53:52.221694 23685 tablet.cc:2369] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:53:52.284324 23854 raft_consensus.cc:482] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:52.284590 23854 raft_consensus.cc:504] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.285436 23854 leader_election.cc:288] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1:44671), 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427)
I20260619 18:53:52.286211 23494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f075cacd484ad58f6bd12f4d8f8cf2" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e073e2c830491a86ae0ee0388a56c8" is_pre_election: true
I20260619 18:53:52.286226 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f075cacd484ad58f6bd12f4d8f8cf2" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4594e77ee70b46cfa03352fb496a3cc6" is_pre_election: true
I20260619 18:53:52.286504 23774 raft_consensus.cc:2454] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 0.
I20260619 18:53:52.286504 23494 raft_consensus.cc:2454] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 0.
I20260619 18:53:52.287038 23567 leader_election.cc:302] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7e073e2c830491a86ae0ee0388a56c8, f3c1da409f4541dbaa0e7b5522886166; no voters:
I20260619 18:53:52.287467 23854 raft_consensus.cc:2781] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:52.287768 23854 raft_consensus.cc:482] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:52.287899 23854 raft_consensus.cc:3037] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.290364 23854 raft_consensus.cc:504] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.291263 23854 leader_election.cc:288] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 election: Requested vote from peers e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1:44671), 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427)
I20260619 18:53:52.291972 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f075cacd484ad58f6bd12f4d8f8cf2" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4594e77ee70b46cfa03352fb496a3cc6"
I20260619 18:53:52.291996 23494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f075cacd484ad58f6bd12f4d8f8cf2" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e073e2c830491a86ae0ee0388a56c8"
I20260619 18:53:52.292243 23494 raft_consensus.cc:3037] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.292243 23774 raft_consensus.cc:3037] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.295753 23774 raft_consensus.cc:2454] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 1.
I20260619 18:53:52.295753 23494 raft_consensus.cc:2454] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 1.
I20260619 18:53:52.296417 23567 leader_election.cc:302] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7e073e2c830491a86ae0ee0388a56c8, f3c1da409f4541dbaa0e7b5522886166; no voters:
I20260619 18:53:52.296842 23854 raft_consensus.cc:2781] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:52.297034 23854 raft_consensus.cc:686] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 1 LEADER]: Becoming Leader. State: Replica: f3c1da409f4541dbaa0e7b5522886166, State: Running, Role: LEADER
I20260619 18:53:52.297338 23854 consensus_queue.cc:237] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.301371 23335 catalog_manager.cc:5495] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2). New cstate: current_term: 1 leader_uuid: "f3c1da409f4541dbaa0e7b5522886166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } health_report { overall_health: UNKNOWN } } }
Jun 19 18:53:52 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895229, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.62@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:52.369843 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.350227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43367 (local address 127.0.0.1:42754)
0619 18:53:52.351433 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:52.351461 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:52.351729 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:52.352303 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:52.352307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:52.353299 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:52.353691 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.353704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.355540 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.355546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.356739 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.356755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.358012 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.358019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.358176 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.359104 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:52.359132 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:52.362117 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:52.366832 (+ 4715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:52.366851 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:52.366859 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:52.367415 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:52.367975 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:52.367979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:52.367982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:52.368144 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:52.368833 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:52.368840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:52.369034 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 18:53:52.369177 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 18:53:52.369457 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":821,"threads_started":1}
I20260619 18:53:52.370196 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.350263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.15.91.62:43367)
0619 18:53:52.350652 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:52.350663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:52.351666 (+ 1003us) server_negotiation.cc:407] Connection header received
0619 18:53:52.351912 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:52.351919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:52.352034 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:52.352211 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:52.353955 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.355258 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.356973 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.357683 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.357884 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.362401 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:52.362410 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:52.362415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:52.362461 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:52.366505 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:52.367591 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:52.367598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:52.367602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:52.367788 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:52.368279 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:52.368286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:52.368289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:52.368658 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:52.368844 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:52.369336 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 18:53:52.369671 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":190}
I20260619 18:53:52.376418 23335 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:42754:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:53:52.376852 23545 tablet.cc:2369] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:53:52.377774 23335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:52.387004 23474 tablet_service.cc:1467] Processing CreateTablet for tablet 5500f0f94ceb407a8824bc1e51ec86df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd22c0b882894269bdc3835be13612cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:53:52.387104 23614 tablet_service.cc:1467] Processing CreateTablet for tablet 5500f0f94ceb407a8824bc1e51ec86df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd22c0b882894269bdc3835be13612cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:53:52.387648 23474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5500f0f94ceb407a8824bc1e51ec86df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:52.387741 23614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5500f0f94ceb407a8824bc1e51ec86df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:52.392222 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 5500f0f94ceb407a8824bc1e51ec86df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd22c0b882894269bdc3835be13612cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:53:52.392912 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5500f0f94ceb407a8824bc1e51ec86df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:52.395682 23848 tablet_bootstrap.cc:492] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: Bootstrap starting.
I20260619 18:53:52.397651 23848 tablet_bootstrap.cc:654] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:52.401805 23850 tablet_bootstrap.cc:492] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6: Bootstrap starting.
I20260619 18:53:52.403874 23850 tablet_bootstrap.cc:654] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:52.406152 23849 tablet_bootstrap.cc:492] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166: Bootstrap starting.
I20260619 18:53:52.408349 23849 tablet_bootstrap.cc:654] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:52.410296 23848 tablet_bootstrap.cc:492] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: No bootstrap required, opened a new log
I20260619 18:53:52.410668 23848 ts_tablet_manager.cc:1397] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260619 18:53:52.411883 23848 raft_consensus.cc:348] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.412385 23848 raft_consensus.cc:374] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.412552 23848 raft_consensus.cc:729] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e073e2c830491a86ae0ee0388a56c8, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.413014 23848 consensus_queue.cc:260] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.413295 23850 tablet_bootstrap.cc:492] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6: No bootstrap required, opened a new log
I20260619 18:53:52.413509 23850 ts_tablet_manager.cc:1397] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260619 18:53:52.414455 23850 raft_consensus.cc:348] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.414745 23850 raft_consensus.cc:374] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.414827 23850 raft_consensus.cc:729] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4594e77ee70b46cfa03352fb496a3cc6, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.415182 23850 consensus_queue.cc:260] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.415716 23856 tablet_replica.cc:439] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } }
I20260619 18:53:52.416035 23856 tablet_replica.cc:442] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:52.416165 23855 tablet_replica.cc:439] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } }
I20260619 18:53:52.416658 23855 tablet_replica.cc:442] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:52.417981 23848 ts_tablet_manager.cc:1428] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260619 18:53:52.418757 23849 tablet_bootstrap.cc:492] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166: No bootstrap required, opened a new log
I20260619 18:53:52.418985 23850 ts_tablet_manager.cc:1428] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260619 18:53:52.418984 23849 ts_tablet_manager.cc:1397] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260619 18:53:52.419790 23849 raft_consensus.cc:348] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.420059 23849 raft_consensus.cc:374] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.420156 23849 raft_consensus.cc:729] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c1da409f4541dbaa0e7b5522886166, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.420446 23849 consensus_queue.cc:260] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.421120 23849 ts_tablet_manager.cc:1428] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 18:53:52.421341 23854 tablet_replica.cc:439] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } }
I20260619 18:53:52.421697 23854 tablet_replica.cc:442] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:52.474902 23855 raft_consensus.cc:482] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:52.475165 23855 raft_consensus.cc:504] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.477226 23855 leader_election.cc:288] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2:44971), 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427)
I20260619 18:53:52.486704 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.478386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:58753 (local address 127.15.91.3:44427)
0619 18:53:52.478628 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:52.478636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:52.478666 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:53:52.478822 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:52.478831 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:52.478978 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:52.479112 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:52.480071 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.481097 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.483987 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.485418 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.485596 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.485724 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:52.486242 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 18:53:52.486386 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 18:53:52.486850 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.477601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:44427 (local address 127.15.91.1:58753)
0619 18:53:52.478440 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:52.478464 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:52.478640 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:52.479308 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:52.479313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:52.479345 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:52.479788 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.479804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.481269 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.481276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.483695 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.483723 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.485722 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.485729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.485975 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.486072 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 18:53:52.486210 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 18:53:52.486370 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":123,"spinlock_wait_cycles":1280,"thread_start_us":2084,"threads_started":1}
I20260619 18:53:52.487370 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.477761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:38637 (local address 127.15.91.2:44971)
0619 18:53:52.478207 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:52.478215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:52.479335 (+ 1120us) server_negotiation.cc:407] Connection header received
0619 18:53:52.479678 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:52.479685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:52.479789 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:52.479870 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:53:52.480732 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.482008 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.484608 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.485578 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.485764 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.485867 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:52.486422 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 18:53:52.486584 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260619 18:53:52.487586 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.477672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:44971 (local address 127.15.91.1:38637)
0619 18:53:52.479226 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:52.479246 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:52.479485 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:52.480086 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:52.480092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:52.480127 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:53:52.480537 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.480551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.482190 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.482196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.484383 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.484398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.485961 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.485972 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.486204 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.486240 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:53:52.487004 (+ 764us) client_negotiation.cc:240] Negotiation successful
0619 18:53:52.487180 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1423,"mutex_wait_us":69,"spinlock_wait_cycles":36992,"thread_start_us":422,"threads_started":1}
I20260619 18:53:52.487627 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5500f0f94ceb407a8824bc1e51ec86df" candidate_uuid: "e7e073e2c830491a86ae0ee0388a56c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4594e77ee70b46cfa03352fb496a3cc6" is_pre_election: true
I20260619 18:53:52.487923 23774 raft_consensus.cc:2454] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7e073e2c830491a86ae0ee0388a56c8 in term 0.
I20260619 18:53:52.488569 23424 leader_election.cc:302] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4594e77ee70b46cfa03352fb496a3cc6, e7e073e2c830491a86ae0ee0388a56c8; no voters:
I20260619 18:53:52.488534 23634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5500f0f94ceb407a8824bc1e51ec86df" candidate_uuid: "e7e073e2c830491a86ae0ee0388a56c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c1da409f4541dbaa0e7b5522886166" is_pre_election: true
I20260619 18:53:52.489004 23855 raft_consensus.cc:2781] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:52.489043 23634 raft_consensus.cc:2454] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7e073e2c830491a86ae0ee0388a56c8 in term 0.
I20260619 18:53:52.489166 23855 raft_consensus.cc:482] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:52.489290 23855 raft_consensus.cc:3037] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.492141 23855 raft_consensus.cc:504] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.493587 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5500f0f94ceb407a8824bc1e51ec86df" candidate_uuid: "e7e073e2c830491a86ae0ee0388a56c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4594e77ee70b46cfa03352fb496a3cc6"
I20260619 18:53:52.493630 23634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5500f0f94ceb407a8824bc1e51ec86df" candidate_uuid: "e7e073e2c830491a86ae0ee0388a56c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c1da409f4541dbaa0e7b5522886166"
I20260619 18:53:52.493881 23774 raft_consensus.cc:3037] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.493891 23634 raft_consensus.cc:3037] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.496515 23774 raft_consensus.cc:2454] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7e073e2c830491a86ae0ee0388a56c8 in term 1.
I20260619 18:53:52.496929 23855 leader_election.cc:288] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [CANDIDATE]: Term 1 election: Requested vote from peers f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2:44971), 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427)
I20260619 18:53:52.497206 23424 leader_election.cc:302] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4594e77ee70b46cfa03352fb496a3cc6, e7e073e2c830491a86ae0ee0388a56c8; no voters:
I20260619 18:53:52.497498 23855 raft_consensus.cc:2781] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:52.497803 23634 raft_consensus.cc:2454] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7e073e2c830491a86ae0ee0388a56c8 in term 1.
I20260619 18:53:52.498158 23855 raft_consensus.cc:686] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [term 1 LEADER]: Becoming Leader. State: Replica: e7e073e2c830491a86ae0ee0388a56c8, State: Running, Role: LEADER
I20260619 18:53:52.498627 23855 consensus_queue.cc:237] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.507078 23889 tablet_replica.cc:439] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: TxnStatusTablet state changed. Reason: New leader e7e073e2c830491a86ae0ee0388a56c8. Latest consensus state: current_term: 1 leader_uuid: "e7e073e2c830491a86ae0ee0388a56c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } }
I20260619 18:53:52.507535 23889 tablet_replica.cc:442] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:53:52.508493 23891 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:53:52.508770 23891 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:53:52.508838 23334 catalog_manager.cc:5495] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1). New cstate: current_term: 1 leader_uuid: "e7e073e2c830491a86ae0ee0388a56c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } health_report { overall_health: UNKNOWN } } }
I20260619 18:53:52.545112 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.536742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.15.91.1:44671)
0619 18:53:52.537294 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:52.537301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:52.537325 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:53:52.537447 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:52.537452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:52.537585 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:52.537786 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:52.538285 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.539602 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.541764 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.542495 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.542672 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.543349 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:52.544253 (+ 904us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:52.544425 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:52.544586 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 18:53:52.544700 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":283,"threads_started":1}
I20260619 18:53:52.545511 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.535526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:44671 (local address 127.0.0.1:34688)
0619 18:53:52.535877 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:52.535901 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:52.536071 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:52.537849 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:52.537855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:52.537872 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:53:52.538104 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.538117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.539777 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.539784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.541523 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.541547 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.542755 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.542764 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.542978 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.543075 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:53:52.544401 (+ 1326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:53:52.544433 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:53:52.544539 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 18:53:52.544745 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 18:53:52.551458 23774 raft_consensus.cc:1261] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Refusing update from remote peer e7e073e2c830491a86ae0ee0388a56c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:52.551921 23634 raft_consensus.cc:1261] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Refusing update from remote peer e7e073e2c830491a86ae0ee0388a56c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:52.552410 23855 consensus_queue.cc:1035] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:52.553004 23889 consensus_queue.cc:1035] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:52.567055 23856 tablet_replica.cc:439] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6: TxnStatusTablet state changed. Reason: New leader e7e073e2c830491a86ae0ee0388a56c8. Latest consensus state: current_term: 1 leader_uuid: "e7e073e2c830491a86ae0ee0388a56c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } }
I20260619 18:53:52.567368 23856 tablet_replica.cc:442] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:52.568076 23854 tablet_replica.cc:439] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166: TxnStatusTablet state changed. Reason: New leader e7e073e2c830491a86ae0ee0388a56c8. Latest consensus state: current_term: 1 leader_uuid: "e7e073e2c830491a86ae0ee0388a56c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } }
I20260619 18:53:52.568460 23854 tablet_replica.cc:442] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:52.571952 23863 mvcc.cc:205] Tried to move back new op lower bound from 7298642872519372800 to 7298642872319692800. Current Snapshot: MvccSnapshot[applied={T|T < 7298642872519372800}]
I20260619 18:53:52.578114 23889 tablet_replica.cc:439] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7e073e2c830491a86ae0ee0388a56c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } }
I20260619 18:53:52.578464 23889 tablet_replica.cc:442] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:53:52.580175 23856 tablet_replica.cc:439] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7e073e2c830491a86ae0ee0388a56c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } }
I20260619 18:53:52.580458 23856 tablet_replica.cc:442] T 5500f0f94ceb407a8824bc1e51ec86df P 4594e77ee70b46cfa03352fb496a3cc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:52.600050 23889 tablet_replica.cc:439] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7e073e2c830491a86ae0ee0388a56c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } }
I20260619 18:53:52.601317 23889 tablet_replica.cc:442] T 5500f0f94ceb407a8824bc1e51ec86df P e7e073e2c830491a86ae0ee0388a56c8: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:53:52.606218 23854 tablet_replica.cc:439] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7e073e2c830491a86ae0ee0388a56c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } } }
I20260619 18:53:52.606585 23854 tablet_replica.cc:442] T 5500f0f94ceb407a8824bc1e51ec86df P f3c1da409f4541dbaa0e7b5522886166: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:53:52.622809 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.613686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:52835 (local address 127.15.91.1:44671)
0619 18:53:52.613903 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:52.613910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:52.614691 (+ 781us) server_negotiation.cc:407] Connection header received
0619 18:53:52.614981 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:52.614988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:52.615093 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:52.615208 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:52.615922 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.617188 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.619936 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.620692 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.620822 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.621728 (+ 906us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:52.621892 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:52.621976 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:52.622242 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 18:53:52.622368 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:53:52.623908 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.613128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:44671 (local address 127.15.91.2:52835)
0619 18:53:52.614511 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:52.614537 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:52.614804 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:52.615264 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:52.615270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:52.615285 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:53:52.615704 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.615719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.617420 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.617427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.619667 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.619694 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.621098 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.621106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.621345 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.621442 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:53:52.622089 (+ 647us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:53:52.622101 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 18:53:52.623167 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0619 18:53:52.623393 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"thread_start_us":902,"threads_started":1}
I20260619 18:53:52.642787 23494 raft_consensus.cc:1261] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Refusing update from remote peer f3c1da409f4541dbaa0e7b5522886166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:52.643570 23859 consensus_queue.cc:1035] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:52.647073 23774 raft_consensus.cc:1261] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Refusing update from remote peer f3c1da409f4541dbaa0e7b5522886166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:52.647977 23859 consensus_queue.cc:1035] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:52.659415 23862 mvcc.cc:205] Tried to move back new op lower bound from 7298642872895229952 to 7298642871492902912. Current Snapshot: MvccSnapshot[applied={T|T < 7298642872895229952}]
I20260619 18:53:52.674479 23864 mvcc.cc:205] Tried to move back new op lower bound from 7298642872895229952 to 7298642871492902912. Current Snapshot: MvccSnapshot[applied={T|T < 7298642872895229952}]
I20260619 18:53:52.711815 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.705031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:39281 (local address 127.15.91.2:44971)
0619 18:53:52.705233 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:52.705240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:52.705276 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:53:52.705346 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:52.705350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:52.705416 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:52.705536 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:52.706325 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.707457 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.709142 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.709824 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.709937 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.710768 (+ 831us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:52.710929 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:52.710999 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:52.711383 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 18:53:52.711496 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:53:52.712636 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.703655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:44971 (local address 127.15.91.1:39281)
0619 18:53:52.704402 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:52.704425 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:52.704582 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:52.705547 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:52.705552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:52.705566 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:53:52.706122 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.706139 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.707634 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.707638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.708941 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:52.708959 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.710181 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.710188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:52.710390 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.710477 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:53:52.711214 (+ 737us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:53:52.711234 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:53:52.712138 (+ 904us) client_negotiation.cc:240] Negotiation successful
0619 18:53:52.712287 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":3473,"threads_started":1}
I20260619 18:53:52.781718 23334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:52.782145 23334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:52.786846 23334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744: Sending DeleteTablet for 3 replicas of tablet a8f075cacd484ad58f6bd12f4d8f8cf2
I20260619 18:53:52.787964 23474 tablet_service.cc:1514] Processing DeleteTablet for tablet a8f075cacd484ad58f6bd12f4d8f8cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:52 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:34684
I20260619 18:53:52.788003 23614 tablet_service.cc:1514] Processing DeleteTablet for tablet a8f075cacd484ad58f6bd12f4d8f8cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:52 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:58350
I20260619 18:53:52.788908 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet a8f075cacd484ad58f6bd12f4d8f8cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:52 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:55542
I20260619 18:53:52.801013 23910 tablet_replica.cc:330] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6: stopping tablet replica
I20260619 18:53:52.801376 23909 tablet_replica.cc:330] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8: stopping tablet replica
I20260619 18:53:52.801685 23909 raft_consensus.cc:2229] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:52.801996 23910 raft_consensus.cc:2229] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:52.802175 23909 raft_consensus.cc:2258] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:52.802294 23910 raft_consensus.cc:2258] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:52.803910 23909 ts_tablet_manager.cc:1905] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:52.805752 23908 tablet_replica.cc:330] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166: stopping tablet replica
I20260619 18:53:52.806234 23908 raft_consensus.cc:2229] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:52.806850 23908 raft_consensus.cc:2258] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:52.807801 23910 ts_tablet_manager.cc:1905] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:52.808483 23908 ts_tablet_manager.cc:1905] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:52.816675 23910 ts_tablet_manager.cc:1918] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:52.817001 23910 log.cc:1196] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/a8f075cacd484ad58f6bd12f4d8f8cf2
I20260619 18:53:52.817677 23910 ts_tablet_manager.cc:1939] T a8f075cacd484ad58f6bd12f4d8f8cf2 P 4594e77ee70b46cfa03352fb496a3cc6: Deleting consensus metadata
I20260619 18:53:52.819209 23321 catalog_manager.cc:4841] TS 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427): tablet a8f075cacd484ad58f6bd12f4d8f8cf2 (table test-table [id=a5d869f3a7554f90b9c414e5bdce3987]) successfully deleted
I20260619 18:53:52.822810 23908 ts_tablet_manager.cc:1918] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:52.823294 23908 log.cc:1196] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/a8f075cacd484ad58f6bd12f4d8f8cf2
I20260619 18:53:52.823261 23909 ts_tablet_manager.cc:1918] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:52.823438 23909 log.cc:1196] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/a8f075cacd484ad58f6bd12f4d8f8cf2
I20260619 18:53:52.824003 23908 ts_tablet_manager.cc:1939] T a8f075cacd484ad58f6bd12f4d8f8cf2 P f3c1da409f4541dbaa0e7b5522886166: Deleting consensus metadata
I20260619 18:53:52.824005 23909 ts_tablet_manager.cc:1939] T a8f075cacd484ad58f6bd12f4d8f8cf2 P e7e073e2c830491a86ae0ee0388a56c8: Deleting consensus metadata
I20260619 18:53:52.825402 23324 catalog_manager.cc:4841] TS f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2:44971): tablet a8f075cacd484ad58f6bd12f4d8f8cf2 (table test-table [id=a5d869f3a7554f90b9c414e5bdce3987]) successfully deleted
I20260619 18:53:52.825719 23324 catalog_manager.cc:4841] TS e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1:44671): tablet a8f075cacd484ad58f6bd12f4d8f8cf2 (table test-table [id=a5d869f3a7554f90b9c414e5bdce3987]) successfully deleted
Jun 19 18:53:52 dist-test-slave-dm3f krb5kdc[23292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:53:52 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:53:52 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:53:52.882684 23711 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:53:52.870806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.15.91.62:43367)
0619 18:53:52.871034 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:52.871041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:52.871519 (+ 478us) server_negotiation.cc:407] Connection header received
0619 18:53:52.871761 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:52.871767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:52.871874 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:52.872001 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:52.873555 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.874460 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.876040 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.876585 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.876757 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.882394 (+ 5637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42762: BlockingRecv error: recv got EOF from 127.0.0.1:42762 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 18:53:52 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM
I20260619 18:53:52.904080 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:52.886508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.15.91.62:43367)
0619 18:53:52.886749 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:52.886757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:52.887101 (+ 344us) server_negotiation.cc:407] Connection header received
0619 18:53:52.887332 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:52.887337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:52.887420 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:52.887551 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:52.888926 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.889737 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.891280 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:52.891742 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:52.891917 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:52.896630 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:52.896640 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:52.896646 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:52.896704 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:52.900492 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:52.901542 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:52.901551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:52.901555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:52.901741 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:52.902246 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:52.902252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:52.902256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:52.902693 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:52.903049 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:52.903368 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 18:53:52.903652 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 18:53:52.908787 23334 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:53:52.909581 23334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:52.918321 23474 tablet_service.cc:1467] Processing CreateTablet for tablet e47fdecb25994d96a0ffd71104f620dd (DEFAULT_TABLE table=test-table [id=ba3a2cf76d0c445abeb5cbacd5bb6500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:52.918421 23614 tablet_service.cc:1467] Processing CreateTablet for tablet e47fdecb25994d96a0ffd71104f620dd (DEFAULT_TABLE table=test-table [id=ba3a2cf76d0c445abeb5cbacd5bb6500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:52.918956 23614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47fdecb25994d96a0ffd71104f620dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:52.918663 23754 tablet_service.cc:1467] Processing CreateTablet for tablet e47fdecb25994d96a0ffd71104f620dd (DEFAULT_TABLE table=test-table [id=ba3a2cf76d0c445abeb5cbacd5bb6500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:52.919041 23474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47fdecb25994d96a0ffd71104f620dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:52.919421 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47fdecb25994d96a0ffd71104f620dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:52.929589 23925 tablet_bootstrap.cc:492] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8: Bootstrap starting.
I20260619 18:53:52.930627 23923 tablet_bootstrap.cc:492] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166: Bootstrap starting.
I20260619 18:53:52.931844 23925 tablet_bootstrap.cc:654] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:52.932456 23923 tablet_bootstrap.cc:654] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:52.934537 23924 tablet_bootstrap.cc:492] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6: Bootstrap starting.
I20260619 18:53:52.936589 23924 tablet_bootstrap.cc:654] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:52.939606 23924 tablet_bootstrap.cc:492] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6: No bootstrap required, opened a new log
I20260619 18:53:52.939808 23924 ts_tablet_manager.cc:1397] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:53:52.940713 23924 raft_consensus.cc:348] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.941013 23924 raft_consensus.cc:374] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.941114 23924 raft_consensus.cc:729] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4594e77ee70b46cfa03352fb496a3cc6, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.941421 23924 consensus_queue.cc:260] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.942348 23925 tablet_bootstrap.cc:492] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8: No bootstrap required, opened a new log
I20260619 18:53:52.942660 23925 ts_tablet_manager.cc:1397] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260619 18:53:52.944048 23925 raft_consensus.cc:348] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.945358 23924 ts_tablet_manager.cc:1428] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:53:52.944712 23925 raft_consensus.cc:374] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.945937 23925 raft_consensus.cc:729] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e073e2c830491a86ae0ee0388a56c8, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.946437 23925 consensus_queue.cc:260] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.947629 23925 ts_tablet_manager.cc:1428] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:53:52.952313 23923 tablet_bootstrap.cc:492] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166: No bootstrap required, opened a new log
I20260619 18:53:52.952554 23923 ts_tablet_manager.cc:1397] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.010s
I20260619 18:53:52.953339 23923 raft_consensus.cc:348] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.953660 23923 raft_consensus.cc:374] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:52.953792 23923 raft_consensus.cc:729] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c1da409f4541dbaa0e7b5522886166, State: Initialized, Role: FOLLOWER
I20260619 18:53:52.954085 23923 consensus_queue.cc:260] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.954947 23923 ts_tablet_manager.cc:1428] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 18:53:52.958863 23855 raft_consensus.cc:482] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:52.959132 23855 raft_consensus.cc:504] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.959965 23855 leader_election.cc:288] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2:44971), 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427)
I20260619 18:53:52.960580 23634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47fdecb25994d96a0ffd71104f620dd" candidate_uuid: "e7e073e2c830491a86ae0ee0388a56c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c1da409f4541dbaa0e7b5522886166" is_pre_election: true
I20260619 18:53:52.960565 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47fdecb25994d96a0ffd71104f620dd" candidate_uuid: "e7e073e2c830491a86ae0ee0388a56c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4594e77ee70b46cfa03352fb496a3cc6" is_pre_election: true
I20260619 18:53:52.960844 23774 raft_consensus.cc:2454] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7e073e2c830491a86ae0ee0388a56c8 in term 0.
I20260619 18:53:52.960844 23634 raft_consensus.cc:2454] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7e073e2c830491a86ae0ee0388a56c8 in term 0.
I20260619 18:53:52.961393 23427 leader_election.cc:302] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7e073e2c830491a86ae0ee0388a56c8, f3c1da409f4541dbaa0e7b5522886166; no voters:
I20260619 18:53:52.962296 23855 raft_consensus.cc:2781] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:52.962458 23855 raft_consensus.cc:482] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:52.962570 23855 raft_consensus.cc:3037] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.965258 23855 raft_consensus.cc:504] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.966207 23855 leader_election.cc:288] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [CANDIDATE]: Term 1 election: Requested vote from peers f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2:44971), 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427)
I20260619 18:53:52.966918 23634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47fdecb25994d96a0ffd71104f620dd" candidate_uuid: "e7e073e2c830491a86ae0ee0388a56c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c1da409f4541dbaa0e7b5522886166"
I20260619 18:53:52.966914 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47fdecb25994d96a0ffd71104f620dd" candidate_uuid: "e7e073e2c830491a86ae0ee0388a56c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4594e77ee70b46cfa03352fb496a3cc6"
I20260619 18:53:52.967159 23634 raft_consensus.cc:3037] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.967157 23774 raft_consensus.cc:3037] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:52.970480 23774 raft_consensus.cc:2454] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7e073e2c830491a86ae0ee0388a56c8 in term 1.
I20260619 18:53:52.970480 23634 raft_consensus.cc:2454] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7e073e2c830491a86ae0ee0388a56c8 in term 1.
I20260619 18:53:52.971467 23424 leader_election.cc:302] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4594e77ee70b46cfa03352fb496a3cc6, e7e073e2c830491a86ae0ee0388a56c8; no voters:
I20260619 18:53:52.971864 23855 raft_consensus.cc:2781] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:52.972313 23855 raft_consensus.cc:686] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 1 LEADER]: Becoming Leader. State: Replica: e7e073e2c830491a86ae0ee0388a56c8, State: Running, Role: LEADER
I20260619 18:53:52.972631 23855 consensus_queue.cc:237] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:52.983206 23336 catalog_manager.cc:5495] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1). New cstate: current_term: 1 leader_uuid: "e7e073e2c830491a86ae0ee0388a56c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } health_report { overall_health: UNKNOWN } } }
I20260619 18:53:53.007674 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:53.000368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.15.91.1:44671)
0619 18:53:53.000624 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:53.000633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:53.000759 (+ 126us) server_negotiation.cc:407] Connection header received
0619 18:53:53.001056 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:53.001072 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:53.001176 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:53.001264 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:53.001841 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:53.002879 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:53.005318 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:53.005844 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:53.005995 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:53.006635 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:53.006854 (+ 219us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:53.006945 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:53.007199 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 18:53:53.007286 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:53:53.011693 23634 raft_consensus.cc:1261] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Refusing update from remote peer e7e073e2c830491a86ae0ee0388a56c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:53.011739 23774 raft_consensus.cc:1261] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Refusing update from remote peer e7e073e2c830491a86ae0ee0388a56c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:53.012493 23897 consensus_queue.cc:1035] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:53.013374 23855 consensus_queue.cc:1035] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:53.035686 23336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:53.036018 23336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:53.039971 23336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744: Sending DeleteTablet for 3 replicas of tablet e47fdecb25994d96a0ffd71104f620dd
I20260619 18:53:53.041118 23474 tablet_service.cc:1514] Processing DeleteTablet for tablet e47fdecb25994d96a0ffd71104f620dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:53 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:34684
I20260619 18:53:53.041366 23614 tablet_service.cc:1514] Processing DeleteTablet for tablet e47fdecb25994d96a0ffd71104f620dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:53 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:58350
I20260619 18:53:53.041674 23908 tablet_replica.cc:330] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166: stopping tablet replica
I20260619 18:53:53.041731 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet e47fdecb25994d96a0ffd71104f620dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:53 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:55542
I20260619 18:53:53.041909 23908 raft_consensus.cc:2229] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:53.042080 23909 tablet_replica.cc:330] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8: stopping tablet replica
I20260619 18:53:53.042187 23908 raft_consensus.cc:2258] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:53.042312 23909 raft_consensus.cc:2229] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:53.042958 23909 raft_consensus.cc:2258] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:53.043310 23910 tablet_replica.cc:330] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6: stopping tablet replica
I20260619 18:53:53.043371 23908 ts_tablet_manager.cc:1905] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:53.043532 23910 raft_consensus.cc:2229] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:53.043771 23910 raft_consensus.cc:2258] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:53.044963 23909 ts_tablet_manager.cc:1905] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:53.045367 23910 ts_tablet_manager.cc:1905] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:53.051661 23908 ts_tablet_manager.cc:1918] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:53.051875 23908 log.cc:1196] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/e47fdecb25994d96a0ffd71104f620dd
I20260619 18:53:53.052340 23908 ts_tablet_manager.cc:1939] T e47fdecb25994d96a0ffd71104f620dd P f3c1da409f4541dbaa0e7b5522886166: Deleting consensus metadata
I20260619 18:53:53.053354 23909 ts_tablet_manager.cc:1918] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:53.053546 23909 log.cc:1196] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/e47fdecb25994d96a0ffd71104f620dd
I20260619 18:53:53.054035 23910 ts_tablet_manager.cc:1918] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:53:53.054033 23324 catalog_manager.cc:4841] TS f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2:44971): tablet e47fdecb25994d96a0ffd71104f620dd (table test-table [id=ba3a2cf76d0c445abeb5cbacd5bb6500]) successfully deleted
I20260619 18:53:53.054509 23910 log.cc:1196] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/e47fdecb25994d96a0ffd71104f620dd
I20260619 18:53:53.054796 23909 ts_tablet_manager.cc:1939] T e47fdecb25994d96a0ffd71104f620dd P e7e073e2c830491a86ae0ee0388a56c8: Deleting consensus metadata
I20260619 18:53:53.055143 23910 ts_tablet_manager.cc:1939] T e47fdecb25994d96a0ffd71104f620dd P 4594e77ee70b46cfa03352fb496a3cc6: Deleting consensus metadata
I20260619 18:53:53.056002 23324 catalog_manager.cc:4841] TS e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1:44671): tablet e47fdecb25994d96a0ffd71104f620dd (table test-table [id=ba3a2cf76d0c445abeb5cbacd5bb6500]) successfully deleted
I20260619 18:53:53.056504 23321 catalog_manager.cc:4841] TS 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427): tablet e47fdecb25994d96a0ffd71104f620dd (table test-table [id=ba3a2cf76d0c445abeb5cbacd5bb6500]) successfully deleted
I20260619 18:53:53.056344 23336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:53:53.057081 23336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:53:53.065398 23474 tablet_service.cc:1467] Processing CreateTablet for tablet 13fe0d373fd9497ab0a33fe5132b4857 (DEFAULT_TABLE table=test-table [id=5d35da04da304b769d9f763b420bc3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:53.065629 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 13fe0d373fd9497ab0a33fe5132b4857 (DEFAULT_TABLE table=test-table [id=5d35da04da304b769d9f763b420bc3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:53.066100 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13fe0d373fd9497ab0a33fe5132b4857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:53.066111 23474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13fe0d373fd9497ab0a33fe5132b4857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:53.071204 23614 tablet_service.cc:1467] Processing CreateTablet for tablet 13fe0d373fd9497ab0a33fe5132b4857 (DEFAULT_TABLE table=test-table [id=5d35da04da304b769d9f763b420bc3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:53:53.071771 23614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13fe0d373fd9497ab0a33fe5132b4857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:53.073298 23924 tablet_bootstrap.cc:492] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6: Bootstrap starting.
I20260619 18:53:53.075606 23924 tablet_bootstrap.cc:654] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:53.075692 23925 tablet_bootstrap.cc:492] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8: Bootstrap starting.
I20260619 18:53:53.077982 23925 tablet_bootstrap.cc:654] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:53.082245 23923 tablet_bootstrap.cc:492] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166: Bootstrap starting.
I20260619 18:53:53.084237 23924 tablet_bootstrap.cc:492] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6: No bootstrap required, opened a new log
I20260619 18:53:53.084533 23924 ts_tablet_manager.cc:1397] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 18:53:53.084829 23923 tablet_bootstrap.cc:654] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:53.086652 23924 raft_consensus.cc:348] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:53.087126 23924 raft_consensus.cc:374] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:53.087250 23924 raft_consensus.cc:729] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4594e77ee70b46cfa03352fb496a3cc6, State: Initialized, Role: FOLLOWER
I20260619 18:53:53.088997 23924 consensus_queue.cc:260] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:53.089812 23924 ts_tablet_manager.cc:1428] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 18:53:53.094615 23925 tablet_bootstrap.cc:492] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8: No bootstrap required, opened a new log
I20260619 18:53:53.094923 23925 ts_tablet_manager.cc:1397] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20260619 18:53:53.095888 23925 raft_consensus.cc:348] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:53.096182 23925 raft_consensus.cc:374] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:53.096269 23925 raft_consensus.cc:729] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e073e2c830491a86ae0ee0388a56c8, State: Initialized, Role: FOLLOWER
I20260619 18:53:53.096593 23925 consensus_queue.cc:260] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:53.097328 23925 ts_tablet_manager.cc:1428] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 18:53:53.099435 23923 tablet_bootstrap.cc:492] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166: No bootstrap required, opened a new log
I20260619 18:53:53.100203 23923 ts_tablet_manager.cc:1397] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.000s
I20260619 18:53:53.101776 23923 raft_consensus.cc:348] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:53.102433 23923 raft_consensus.cc:374] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:53.102548 23923 raft_consensus.cc:729] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c1da409f4541dbaa0e7b5522886166, State: Initialized, Role: FOLLOWER
I20260619 18:53:53.102975 23923 consensus_queue.cc:260] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:53.104631 23923 ts_tablet_manager.cc:1428] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 18:53:53.252025 23859 raft_consensus.cc:482] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:53:53.252280 23859 raft_consensus.cc:504] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:53.253031 23859 leader_election.cc:288] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1:44671), 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427)
I20260619 18:53:53.253779 23494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13fe0d373fd9497ab0a33fe5132b4857" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e073e2c830491a86ae0ee0388a56c8" is_pre_election: true
I20260619 18:53:53.253826 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13fe0d373fd9497ab0a33fe5132b4857" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4594e77ee70b46cfa03352fb496a3cc6" is_pre_election: true
I20260619 18:53:53.254067 23774 raft_consensus.cc:2454] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 0.
I20260619 18:53:53.254076 23494 raft_consensus.cc:2454] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 0.
I20260619 18:53:53.254841 23567 leader_election.cc:302] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7e073e2c830491a86ae0ee0388a56c8, f3c1da409f4541dbaa0e7b5522886166; no voters:
I20260619 18:53:53.255180 23859 raft_consensus.cc:2781] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:53:53.255285 23859 raft_consensus.cc:482] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:53:53.255398 23859 raft_consensus.cc:3037] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:53.257633 23859 raft_consensus.cc:504] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:53.259092 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13fe0d373fd9497ab0a33fe5132b4857" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4594e77ee70b46cfa03352fb496a3cc6"
I20260619 18:53:53.259053 23494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13fe0d373fd9497ab0a33fe5132b4857" candidate_uuid: "f3c1da409f4541dbaa0e7b5522886166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e073e2c830491a86ae0ee0388a56c8"
I20260619 18:53:53.259338 23774 raft_consensus.cc:3037] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:53.259356 23494 raft_consensus.cc:3037] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:53.262876 23494 raft_consensus.cc:2454] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 1.
I20260619 18:53:53.263005 23774 raft_consensus.cc:2454] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c1da409f4541dbaa0e7b5522886166 in term 1.
I20260619 18:53:53.263590 23564 leader_election.cc:302] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4594e77ee70b46cfa03352fb496a3cc6, e7e073e2c830491a86ae0ee0388a56c8, f3c1da409f4541dbaa0e7b5522886166; no voters:
I20260619 18:53:53.263875 23859 leader_election.cc:288] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [CANDIDATE]: Term 1 election: Requested vote from peers e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1:44671), 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427)
I20260619 18:53:53.264109 23859 raft_consensus.cc:2781] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:53.264437 23859 raft_consensus.cc:686] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 1 LEADER]: Becoming Leader. State: Replica: f3c1da409f4541dbaa0e7b5522886166, State: Running, Role: LEADER
I20260619 18:53:53.264912 23859 consensus_queue.cc:237] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } }
I20260619 18:53:53.270364 23337 catalog_manager.cc:5495] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2). New cstate: current_term: 1 leader_uuid: "f3c1da409f4541dbaa0e7b5522886166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c1da409f4541dbaa0e7b5522886166" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 } health_report { overall_health: UNKNOWN } } }
I20260619 18:53:53.340335 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:53.334338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.15.91.2:44971)
0619 18:53:53.335181 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:53.335186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:53.335204 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:53:53.335267 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:53.335271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:53.335343 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:53.335474 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:53:53.335930 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:53.336694 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:53.338195 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:53.338641 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:53.338864 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:53.339444 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:53:53.339573 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:53:53.339692 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:53.339884 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 18:53:53.339976 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":553,"threads_started":1}
I20260619 18:53:53.353348 23494 raft_consensus.cc:1261] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Refusing update from remote peer f3c1da409f4541dbaa0e7b5522886166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:53.353410 23774 raft_consensus.cc:1261] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Refusing update from remote peer f3c1da409f4541dbaa0e7b5522886166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:53:53.354472 23936 consensus_queue.cc:1035] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4594e77ee70b46cfa03352fb496a3cc6" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:53.355083 23859 consensus_queue.cc:1035] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e073e2c830491a86ae0ee0388a56c8" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:53:53.442401 23337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:53.442696 23337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:53:53.446329 23337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc44ddc229764208a446506651b16744: Sending DeleteTablet for 3 replicas of tablet 13fe0d373fd9497ab0a33fe5132b4857
I20260619 18:53:53.447523 23614 tablet_service.cc:1514] Processing DeleteTablet for tablet 13fe0d373fd9497ab0a33fe5132b4857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:53 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:58350
I20260619 18:53:53.447532 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet 13fe0d373fd9497ab0a33fe5132b4857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:53 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:55542
I20260619 18:53:53.447902 23908 tablet_replica.cc:330] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166: stopping tablet replica
I20260619 18:53:53.447911 23910 tablet_replica.cc:330] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6: stopping tablet replica
I20260619 18:53:53.448108 23910 raft_consensus.cc:2229] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:53.448356 23910 raft_consensus.cc:2258] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:53.448113 23908 raft_consensus.cc:2229] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:53:53.449103 23908 raft_consensus.cc:2258] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:53.449652 23910 ts_tablet_manager.cc:1905] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:53.450479 23908 ts_tablet_manager.cc:1905] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:53:53.450960 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 23408
I20260619 18:53:53.457106 23910 ts_tablet_manager.cc:1918] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:53.457310 23910 log.cc:1196] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/13fe0d373fd9497ab0a33fe5132b4857
I20260619 18:53:53.457556 23474 tablet_service.cc:1514] Processing DeleteTablet for tablet 13fe0d373fd9497ab0a33fe5132b4857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:53:53 UTC) from {username='oryx', principal='oryx/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:34684
I20260619 18:53:53.457814 23910 ts_tablet_manager.cc:1939] T 13fe0d373fd9497ab0a33fe5132b4857 P 4594e77ee70b46cfa03352fb496a3cc6: Deleting consensus metadata
I20260619 18:53:53.457948 23909 tablet_replica.cc:330] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8: stopping tablet replica
I20260619 18:53:53.458154 23909 raft_consensus.cc:2229] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:53:53.458395 23909 raft_consensus.cc:2258] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:53:53.459185 23321 catalog_manager.cc:4841] TS 4594e77ee70b46cfa03352fb496a3cc6 (127.15.91.3:44427): tablet 13fe0d373fd9497ab0a33fe5132b4857 (table test-table [id=5d35da04da304b769d9f763b420bc3b1]) successfully deleted
I20260619 18:53:53.460212 23909 ts_tablet_manager.cc:1905] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:53:53 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.91.1@KRBTEST.COM
I20260619 18:53:53.466841 23908 ts_tablet_manager.cc:1918] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:53.467051 23908 log.cc:1196] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/13fe0d373fd9497ab0a33fe5132b4857
I20260619 18:53:53.467519 23908 ts_tablet_manager.cc:1939] T 13fe0d373fd9497ab0a33fe5132b4857 P f3c1da409f4541dbaa0e7b5522886166: Deleting consensus metadata
I20260619 18:53:53.468361 23909 ts_tablet_manager.cc:1918] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:53:53.468555 23909 log.cc:1196] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/13fe0d373fd9497ab0a33fe5132b4857
I20260619 18:53:53.469107 23909 ts_tablet_manager.cc:1939] T 13fe0d373fd9497ab0a33fe5132b4857 P e7e073e2c830491a86ae0ee0388a56c8: Deleting consensus metadata
I20260619 18:53:53.469324 23324 catalog_manager.cc:4841] TS f3c1da409f4541dbaa0e7b5522886166 (127.15.91.2:44971): tablet 13fe0d373fd9497ab0a33fe5132b4857 (table test-table [id=5d35da04da304b769d9f763b420bc3b1]) successfully deleted
I20260619 18:53:53.470386 23324 catalog_manager.cc:4841] TS e7e073e2c830491a86ae0ee0388a56c8 (127.15.91.1:44671): tablet 13fe0d373fd9497ab0a33fe5132b4857 (table test-table [id=5d35da04da304b769d9f763b420bc3b1]) successfully deleted
I20260619 18:53:53.476416 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:53.452914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.15.91.1:44671)
0619 18:53:53.453148 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:53.453155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:53.453174 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:53:53.453236 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:53.453241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:53.453321 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:53.453434 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:53.455351 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:53.456587 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:53.459024 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:53.460497 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:53.460681 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:53.466553 (+ 5872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:53.466578 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:53.466614 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:53.466666 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:53.471417 (+ 4751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:53.472738 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:53.472750 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:53.472753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:53.473034 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:53.473864 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:53.473871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:53.473876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:53.474494 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:53.474886 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:53.475652 (+ 766us) server_negotiation.cc:299] Negotiation successful
0619 18:53:53.475947 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:53:53.477353 23539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:53.640404 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 23408
I20260619 18:53:53.667218 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 23548
Jun 19 18:53:53 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.91.2@KRBTEST.COM
I20260619 18:53:53.684265 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:53.668169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.15.91.2:44971)
0619 18:53:53.668343 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:53.668347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:53.668366 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:53:53.668509 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:53.668513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:53.668580 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:53.668685 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:53.670025 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:53.670847 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:53.672304 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:53.672743 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:53.672869 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:53.677859 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:53.677878 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:53.677894 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:53.677942 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:53.681185 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:53.682137 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:53.682145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:53.682148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:53.682330 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:53.682765 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:53.682774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:53.682780 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:53.683230 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:53.683497 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:53.683774 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 18:53:53.683994 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:53:53.685122 23679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:53.831037 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 23548
I20260619 18:53:53.858866 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 23688
Jun 19 18:53:53 dist-test-slave-dm3f krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.91.3@KRBTEST.COM
I20260619 18:53:53.878207 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:53.859779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.15.91.3:44427)
0619 18:53:53.860442 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:53.860447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:53.860465 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:53:53.860519 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:53.860523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:53.860604 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:53.860746 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:53.862203 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:53.863102 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:53.864516 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:53.865097 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:53.865252 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:53.871388 (+ 6136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:53.871408 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:53.871425 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:53.871471 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:53.874788 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:53.875767 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:53.875776 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:53.875778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:53.875939 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:53.876411 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:53.876416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:53.876420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:53.876846 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:53.877236 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:53.877571 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 18:53:53.877826 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":410,"threads_started":1}
I20260619 18:53:53.878984 23819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:53.978140 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 23688
I20260619 18:53:53.997151 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 23305
I20260619 18:53:53.998786 23377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:54.120555 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 23305
2026-06-19T18:53:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6085 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:53:54 dist-test-slave-dm3f krb5kdc[23954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:53:54 dist-test-slave-dm3f krb5kdc[23954](info): set up 2 sockets
Jun 19 18:53:54 dist-test-slave-dm3f krb5kdc[23954](info): commencing operation
krb5kdc: starting...
W20260619 18:53:55.244417 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:53:55 dist-test-slave-dm3f krb5kdc[23954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:53:55.499770 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:45991
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:45991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:55.881016 23967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:55.881424 23967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:55.881616 23967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:55.893419 23967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:55.893608 23967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:55.893714 23967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:55.893817 23967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:55.893910 23967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:55.905115 23967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:45991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:45991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:55.906572 23967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:55.908139 23967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:55.921609 23973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:55.921559 23972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:55.921653 23975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:55.922947 23967 server_base.cc:1034] running on GCE node
I20260619 18:53:55.923964 23967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:55.925817 23967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:55.927377 23967 hybrid_clock.cc:648] HybridClock initialized: now 1781895235927273 us; error 93 us; skew 500 ppm
I20260619 18:53:55.927917 23967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T18:53:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1878 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:53:56 dist-test-slave-dm3f krb5kdc[23981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:53:56 dist-test-slave-dm3f krb5kdc[23981](info): set up 2 sockets
Jun 19 18:53:56 dist-test-slave-dm3f krb5kdc[23981](info): commencing operation
krb5kdc: starting...
W20260619 18:53:57.124444 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:53:57 dist-test-slave-dm3f krb5kdc[23981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:53:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:53:57.373131 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:44637
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:44637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:57.735683 23994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:57.736023 23994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:57.736195 23994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:57.744060 23994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:53:57.744201 23994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:57.744287 23994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:57.744382 23994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:53:57.744475 23994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:53:57.755167 23994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:44637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:44637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:57.756731 23994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:57.758201 23994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:57.772944 23999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:57.773025 24002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:57.773025 24000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:57.774072 23994 server_base.cc:1034] running on GCE node
I20260619 18:53:57.775141 23994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:57.776937 23994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:57.778201 23994 hybrid_clock.cc:648] HybridClock initialized: now 1781895237778152 us; error 52 us; skew 500 ppm
I20260619 18:53:57.778640 23994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:57 dist-test-slave-dm3f krb5kdc[23981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895237, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:57.782327 23994 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:53:57.784476 23994 webserver.cc:460] Webserver started at http://127.15.91.62:46451/ using document root <none> and password file <none>
I20260619 18:53:57.785696 23994 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:57.785897 23994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:57.786343 23994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:57.790179 23994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c"
format_stamp: "Formatted at 2026-06-19 18:53:57 on dist-test-slave-dm3f"
server_key: "fa42f25b0e0a91554839b28a91716ff3"
server_key_iv: "c3c5958a38ba4a7c8d61b68ce348da60"
server_key_version: "encryptionkey@0"
I20260619 18:53:57.791287 23994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c"
format_stamp: "Formatted at 2026-06-19 18:53:57 on dist-test-slave-dm3f"
server_key: "fa42f25b0e0a91554839b28a91716ff3"
server_key_iv: "c3c5958a38ba4a7c8d61b68ce348da60"
server_key_version: "encryptionkey@0"
I20260619 18:53:57.797776 23994 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:53:57.802258 24009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:57.804088 23994 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:53:57.804378 23994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c"
format_stamp: "Formatted at 2026-06-19 18:53:57 on dist-test-slave-dm3f"
server_key: "fa42f25b0e0a91554839b28a91716ff3"
server_key_iv: "c3c5958a38ba4a7c8d61b68ce348da60"
server_key_version: "encryptionkey@0"
I20260619 18:53:57.804636 23994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:57.830685 23994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:57.835798 23994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:57.836177 23994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:57.857293 23994 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:44637
I20260619 18:53:57.857326 24060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:44637 every 8 connection(s)
I20260619 18:53:57.858985 23994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:53:57.862156 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 23994
I20260619 18:53:57.862406 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:53:57.863190 15724 external_mini_cluster.cc:1428] Setting key d068d8712420bb7f621398a0bb5b45d9
I20260619 18:53:57.863871 24061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:53:57.878479 24061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c: Bootstrap starting.
Jun 19 18:53:57 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:57.882400 24061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c: Neither blocks nor log segments found. Creating new log.
I20260619 18:53:57.883761 24061 log.cc:824] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:53:57.887300 24061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c: No bootstrap required, opened a new log
I20260619 18:53:57.889850 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:57.865509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.15.91.62:44637)
0619 18:53:57.866478 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:57.866492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:57.866544 (+ 52us) server_negotiation.cc:407] Connection header received
0619 18:53:57.869275 (+ 2731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:57.869310 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:57.869886 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:57.870421 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:57.872238 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:57.873699 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:57.875293 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:57.876191 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:57.876528 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:57.881142 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:57.881151 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:57.881169 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:57.881255 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:57.885103 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:57.886017 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:57.886026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:57.886033 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:57.886243 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:57.886692 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:57.886696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:57.886699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:57.887324 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:57.887671 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:57.888199 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 18:53:57.888653 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":439,"threads_started":1}
I20260619 18:53:57.896131 24061 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } }
I20260619 18:53:57.896622 24061 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:53:57.896757 24061 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d5e74531dd419aa6ddfc9bd4f32a3c, State: Initialized, Role: FOLLOWER
I20260619 18:53:57.897408 24061 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } }
I20260619 18:53:57.897686 24061 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:53:57.897835 24061 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:53:57.898025 24061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:53:57.900552 24061 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } }
I20260619 18:53:57.901212 24061 leader_election.cc:302] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9d5e74531dd419aa6ddfc9bd4f32a3c; no voters:
I20260619 18:53:57.901860 24061 leader_election.cc:288] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:53:57.902035 24066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:53:57.902842 24066 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 1 LEADER]: Becoming Leader. State: Replica: a9d5e74531dd419aa6ddfc9bd4f32a3c, State: Running, Role: LEADER
I20260619 18:53:57.903440 24066 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } }
I20260619 18:53:57.904227 24061 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:53:57.906635 24067 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } } }
I20260619 18:53:57.907109 24067 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:57.908236 24071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:53:57.908118 24068 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9d5e74531dd419aa6ddfc9bd4f32a3c. Latest consensus state: current_term: 1 leader_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } } }
I20260619 18:53:57.908490 24068 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [sys.catalog]: This master's current role is: LEADER
I20260619 18:53:57.916193 24071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:53:57.928196 24071 catalog_manager.cc:1347] Generated new cluster ID: 5b3ec0bff7eb4447bbebb1f30177315d
I20260619 18:53:57.928345 24071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:53:57.942360 24071 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:53:57.944451 24071 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:53:57.956673 24071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c: Generated new TSK 0
I20260619 18:53:57.957973 24071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:53:58.023213 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44637
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:58.396824 24087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:58.397207 24087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:58.397390 24087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:58.405480 24087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:58.405663 24087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:58.405864 24087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:53:58.417034 24087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:58.418519 24087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:58.420023 24087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:58.434962 24093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:58.435043 24092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:58.435022 24095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:58.436589 24087 server_base.cc:1034] running on GCE node
I20260619 18:53:58.437331 24087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:58.438434 24087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:58.439708 24087 hybrid_clock.cc:648] HybridClock initialized: now 1781895238439619 us; error 84 us; skew 500 ppm
I20260619 18:53:58.440097 24087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:58 dist-test-slave-dm3f krb5kdc[23981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895238, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:58.443740 24087 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:53:58.445588 24087 webserver.cc:460] Webserver started at http://127.15.91.1:36951/ using document root <none> and password file <none>
I20260619 18:53:58.446422 24087 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:58.446595 24087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:58.447012 24087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:58.450285 24087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "c503a0f785684abfb40d7a2a53c13bee"
format_stamp: "Formatted at 2026-06-19 18:53:58 on dist-test-slave-dm3f"
server_key: "931887295d3a3ea3e26d58163b8bbdec"
server_key_iv: "88a12cf40aa24043679547fda9a033a3"
server_key_version: "encryptionkey@0"
I20260619 18:53:58.451260 24087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "c503a0f785684abfb40d7a2a53c13bee"
format_stamp: "Formatted at 2026-06-19 18:53:58 on dist-test-slave-dm3f"
server_key: "931887295d3a3ea3e26d58163b8bbdec"
server_key_iv: "88a12cf40aa24043679547fda9a033a3"
server_key_version: "encryptionkey@0"
I20260619 18:53:58.457427 24087 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:53:58.461724 24102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:58.463485 24087 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:53:58.463711 24087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "c503a0f785684abfb40d7a2a53c13bee"
format_stamp: "Formatted at 2026-06-19 18:53:58 on dist-test-slave-dm3f"
server_key: "931887295d3a3ea3e26d58163b8bbdec"
server_key_iv: "88a12cf40aa24043679547fda9a033a3"
server_key_version: "encryptionkey@0"
I20260619 18:53:58.463955 24087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:58.476758 24087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:58.481515 24087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:58.481876 24087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:58.483089 24087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:58.484724 24087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:58.484818 24087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:58.484913 24087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:58.484958 24087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:58.523149 24087 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:38191
I20260619 18:53:58.523200 24214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:38191 every 8 connection(s)
I20260619 18:53:58.524706 24087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:53:58.533381 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 24087
I20260619 18:53:58.533618 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:53:58.534456 15724 external_mini_cluster.cc:1428] Setting key b932ad0377101489c847723c11a197c6
Jun 19 18:53:58 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895238, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:58.553117 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:58.527950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:58271 (local address 127.15.91.62:44637)
0619 18:53:58.528470 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:58.528475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:58.529599 (+ 1124us) server_negotiation.cc:407] Connection header received
0619 18:53:58.531893 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:58.531899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:58.532014 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:58.532151 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:58.534904 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:58.536025 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:58.538842 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:58.539355 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:58.539506 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:58.544761 (+ 5255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:58.544772 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:58.544781 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:58.544840 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:58.547959 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:58.549430 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:58.549438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:58.549441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:58.549728 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:58.550669 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:58.550675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:58.550678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:58.551174 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:58.551334 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:58.552598 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0619 18:53:58.552827 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":295,"threads_started":1}
I20260619 18:53:58.554919 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:58.528479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:58271)
0619 18:53:58.529366 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:58.529424 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:58.531488 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:58.532461 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:58.532471 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:58.533380 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:58.534440 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:58.534478 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:58.536268 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:58.536273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:58.538475 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:58.538506 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:58.539711 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:58.539722 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:58.540182 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:58.541157 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:58.541198 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:58.544397 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:58.548229 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:58.548245 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:58.548270 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:58.549147 (+ 877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:58.550014 (+ 867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:58.550024 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:58.550030 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:58.550437 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:58.551416 (+ 979us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:58.551430 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:58.552057 (+ 627us) client_negotiation.cc:769] Sending connection context
0619 18:53:58.552446 (+ 389us) client_negotiation.cc:240] Negotiation successful
0619 18:53:58.553140 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":357,"threads_started":1}
I20260619 18:53:58.557067 24215 heartbeater.cc:347] Connected to a master server at 127.15.91.62:44637
I20260619 18:53:58.557595 24215 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:58.558470 24215 heartbeater.cc:511] Master 127.15.91.62:44637 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:53:58.561368 24026 ts_manager.cc:194] Registered new tserver with Master: c503a0f785684abfb40d7a2a53c13bee (127.15.91.1:38191)
I20260619 18:53:58.563241 24026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:58271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:53:58.595750 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44637
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:58.957248 24221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:58.957635 24221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:58.957816 24221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:58.965857 24221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:58.966055 24221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:58.966251 24221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:53:58.976869 24221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:58.978401 24221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:58.979882 24221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:58.994479 24227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:58.994575 24226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:58.994592 24229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:58.995482 24221 server_base.cc:1034] running on GCE node
I20260619 18:53:58.996272 24221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:58.997557 24221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:58.998868 24221 hybrid_clock.cc:648] HybridClock initialized: now 1781895238998784 us; error 63 us; skew 500 ppm
I20260619 18:53:58.999277 24221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:58 dist-test-slave-dm3f krb5kdc[23981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895238, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:59.003000 24221 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:53:59.004973 24221 webserver.cc:460] Webserver started at http://127.15.91.2:37413/ using document root <none> and password file <none>
I20260619 18:53:59.005868 24221 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:59.006001 24221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:59.006278 24221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:59.010211 24221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "adc328dc575044c99e33f1ab3ce7970a"
format_stamp: "Formatted at 2026-06-19 18:53:59 on dist-test-slave-dm3f"
server_key: "cef32c5f79c8dc7f21b517645f0629c4"
server_key_iv: "c262f9e47b9350f2de062905f495ba0f"
server_key_version: "encryptionkey@0"
I20260619 18:53:59.011281 24221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "adc328dc575044c99e33f1ab3ce7970a"
format_stamp: "Formatted at 2026-06-19 18:53:59 on dist-test-slave-dm3f"
server_key: "cef32c5f79c8dc7f21b517645f0629c4"
server_key_iv: "c262f9e47b9350f2de062905f495ba0f"
server_key_version: "encryptionkey@0"
I20260619 18:53:59.017933 24221 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 18:53:59.022573 24236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:59.024556 24221 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:53:59.024771 24221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "adc328dc575044c99e33f1ab3ce7970a"
format_stamp: "Formatted at 2026-06-19 18:53:59 on dist-test-slave-dm3f"
server_key: "cef32c5f79c8dc7f21b517645f0629c4"
server_key_iv: "c262f9e47b9350f2de062905f495ba0f"
server_key_version: "encryptionkey@0"
I20260619 18:53:59.024982 24221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:59.045032 24221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:59.049973 24221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:59.050268 24221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:59.051345 24221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:59.052850 24221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:59.052935 24221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:59.053035 24221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:59.053081 24221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:59.091485 24221 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:38217
I20260619 18:53:59.091522 24348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:38217 every 8 connection(s)
I20260619 18:53:59.093060 24221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:53:59.101290 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 24221
I20260619 18:53:59.101513 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:53:59.102035 15724 external_mini_cluster.cc:1428] Setting key e4d9067553e2f6550b9f3d4e752c03ee
Jun 19 18:53:59 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895238, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:59.118693 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:59.096279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:51429 (local address 127.15.91.62:44637)
0619 18:53:59.096860 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:59.096864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:59.098029 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 18:53:59.100294 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:59.100301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:59.100413 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:59.100546 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:59.102651 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:59.103676 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:59.105803 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:59.106277 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:59.106440 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:59.110916 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:59.110922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:59.110926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:59.110976 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:59.114274 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:59.115691 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:59.115696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:59.115699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:59.115874 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:59.116574 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:59.116579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:59.116581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:59.116899 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:59.117033 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:59.118131 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0619 18:53:59.118384 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":462,"thread_start_us":288,"threads_started":1}
I20260619 18:53:59.120436 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:59.096934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:51429)
0619 18:53:59.097812 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:59.097872 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:59.099903 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:59.100857 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:59.100868 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:59.101641 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:59.102405 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:59.102432 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:59.103892 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:59.103897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:59.105595 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:59.105617 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:59.106567 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:59.106576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:59.106911 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:59.107776 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:59.107808 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:59.110533 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:59.114544 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:59.114560 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:59.114586 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:59.115482 (+ 896us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:59.116043 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:59.116051 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:59.116057 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:59.116383 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:59.117063 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:59.117076 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:59.117643 (+ 567us) client_negotiation.cc:769] Sending connection context
0619 18:53:59.117998 (+ 355us) client_negotiation.cc:240] Negotiation successful
0619 18:53:59.118691 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":353,"threads_started":1}
I20260619 18:53:59.122350 24349 heartbeater.cc:347] Connected to a master server at 127.15.91.62:44637
I20260619 18:53:59.122803 24349 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:59.123910 24349 heartbeater.cc:511] Master 127.15.91.62:44637 requested a full tablet report, sending...
I20260619 18:53:59.126050 24026 ts_manager.cc:194] Registered new tserver with Master: adc328dc575044c99e33f1ab3ce7970a (127.15.91.2:38217)
I20260619 18:53:59.127074 24026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:51429
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:53:59.159214 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44637
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:53:59.527848 24355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:53:59.528151 24355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:53:59.528339 24355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:53:59.536496 24355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:53:59.536648 24355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:53:59.536836 24355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:53:59.547264 24355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:53:59.548718 24355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:53:59.550175 24355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:53:59.565294 24360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:59.565218 24361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:53:59.565243 24363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:53:59.566540 24355 server_base.cc:1034] running on GCE node
I20260619 18:53:59.567348 24355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:53:59.568661 24355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:53:59.568817 24215 heartbeater.cc:503] Master 127.15.91.62:44637 was elected leader, sending a full tablet report...
I20260619 18:53:59.569943 24355 hybrid_clock.cc:648] HybridClock initialized: now 1781895239569806 us; error 153 us; skew 500 ppm
I20260619 18:53:59.570334 24355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:53:59 dist-test-slave-dm3f krb5kdc[23981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895239, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:53:59.573940 24355 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:53:59.575755 24355 webserver.cc:460] Webserver started at http://127.15.91.3:38155/ using document root <none> and password file <none>
I20260619 18:53:59.576557 24355 fs_manager.cc:359] Metadata directory not provided
I20260619 18:53:59.576685 24355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:53:59.576987 24355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:53:59.580336 24355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "acd14c7bedd049fea95200cf9b676d67"
format_stamp: "Formatted at 2026-06-19 18:53:59 on dist-test-slave-dm3f"
server_key: "592189c2908b8c4ce273f3bb8ed11652"
server_key_iv: "52a5b7a3b1d6dcb4883b1a4858313334"
server_key_version: "encryptionkey@0"
I20260619 18:53:59.581244 24355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "acd14c7bedd049fea95200cf9b676d67"
format_stamp: "Formatted at 2026-06-19 18:53:59 on dist-test-slave-dm3f"
server_key: "592189c2908b8c4ce273f3bb8ed11652"
server_key_iv: "52a5b7a3b1d6dcb4883b1a4858313334"
server_key_version: "encryptionkey@0"
I20260619 18:53:59.588035 24355 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:53:59.592307 24370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:59.594105 24355 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:53:59.594362 24355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "acd14c7bedd049fea95200cf9b676d67"
format_stamp: "Formatted at 2026-06-19 18:53:59 on dist-test-slave-dm3f"
server_key: "592189c2908b8c4ce273f3bb8ed11652"
server_key_iv: "52a5b7a3b1d6dcb4883b1a4858313334"
server_key_version: "encryptionkey@0"
I20260619 18:53:59.594628 24355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:53:59.621234 24355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:53:59.627202 24355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:53:59.627601 24355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:53:59.629125 24355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:53:59.630874 24355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:53:59.630965 24355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:59.631134 24355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:53:59.631187 24355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:53:59.669610 24355 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:45513
I20260619 18:53:59.669647 24482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:45513 every 8 connection(s)
I20260619 18:53:59.671133 24355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:53:59.675060 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 24355
I20260619 18:53:59.675269 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:53:59.675890 15724 external_mini_cluster.cc:1428] Setting key 730ba3e8baa1a666c859d991a4fb3c78
Jun 19 18:53:59 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895239, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:53:59.699968 24485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:59.674041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:40703 (local address 127.15.91.62:44637)
0619 18:53:59.674600 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:59.674606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:59.676187 (+ 1581us) server_negotiation.cc:407] Connection header received
0619 18:53:59.678876 (+ 2689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:59.678881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:59.678954 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:59.679054 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:59.681058 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:59.681877 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:59.683817 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:59.684245 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:59.684364 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:59.691840 (+ 7476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:59.691847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:59.691852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:59.691903 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:59.695052 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:59.696679 (+ 1627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:59.696685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:59.696688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:59.696904 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:59.697728 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:59.697733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:59.697736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:59.698067 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:59.698220 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:59.699261 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0619 18:53:59.699582 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":297,"threads_started":1}
I20260619 18:53:59.700889 24486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:59.674948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:40703)
0619 18:53:59.675925 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0619 18:53:59.676015 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0619 18:53:59.678513 (+ 2498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:53:59.679459 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:53:59.679475 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:53:59.680139 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:53:59.680814 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:59.680837 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:59.682085 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:59.682090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:59.683605 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:53:59.683624 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:53:59.684491 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:53:59.684497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:53:59.684733 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:59.688614 (+ 3881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:53:59.688647 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:53:59.691528 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:53:59.695389 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:59.695403 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:59.695427 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:59.696370 (+ 943us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:59.697182 (+ 812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:53:59.697188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:53:59.697192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:53:59.697539 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:53:59.698218 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:53:59.698227 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:53:59.698650 (+ 423us) client_negotiation.cc:769] Sending connection context
0619 18:53:59.699081 (+ 431us) client_negotiation.cc:240] Negotiation successful
0619 18:53:59.699594 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":415,"threads_started":1}
I20260619 18:53:59.702663 24483 heartbeater.cc:347] Connected to a master server at 127.15.91.62:44637
I20260619 18:53:59.703146 24483 heartbeater.cc:464] Registering TS with master...
I20260619 18:53:59.704159 24483 heartbeater.cc:511] Master 127.15.91.62:44637 requested a full tablet report, sending...
I20260619 18:53:59.705996 24026 ts_manager.cc:194] Registered new tserver with Master: acd14c7bedd049fea95200cf9b676d67 (127.15.91.3:45513)
I20260619 18:53:59.706935 24026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:40703
I20260619 18:53:59.718827 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab.
Entry for principal oryx/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab.
WARNING: no policy specified for oryx/127.15.91.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab.
Entry for principal oryx/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab.
WARNING: no policy specified for oryx/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.91.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab.
Entry for principal oryx/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab.
I20260619 18:53:59.879102 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 24087
Jun 19 18:53:59 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:53:59.898317 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:53:59.880441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.15.91.1:38191)
0619 18:53:59.881084 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0619 18:53:59.881092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:53:59.881123 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:53:59.881186 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:53:59.881193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:53:59.881421 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:53:59.881574 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:53:59.883672 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:59.884768 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:59.886293 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:53:59.887019 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:53:59.887215 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:53:59.892028 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:53:59.892045 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:53:59.892064 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:53:59.892116 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:53:59.895229 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:59.896103 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:59.896112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:59.896115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:59.896279 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:53:59.896704 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:53:59.896707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:53:59.896709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:53:59.897109 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:53:59.897357 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 18:53:59.897638 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 18:53:59.897946 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":311,"threads_started":1}
I20260619 18:53:59.899917 24210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:53:59.999406 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 24087
I20260619 18:54:00.017899 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 24221
Jun 19 18:54:00 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:54:00.036990 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:00.018916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.15.91.2:38217)
0619 18:54:00.019685 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:00.019693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:00.019722 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:54:00.019785 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:00.019791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:00.020025 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:00.020163 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:00.021442 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:00.022459 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:00.023977 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:00.024627 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:00.024760 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:00.029483 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:00.029501 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:00.029525 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:00.029571 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:00.032924 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:00.033847 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:00.033855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:00.033858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:00.034008 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:00.034963 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:00.034969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:00.034972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:00.035438 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:00.035841 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:00.036259 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 18:54:00.036593 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":487,"threads_started":1}
I20260619 18:54:00.038564 24344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:00.123740 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 24221
I20260619 18:54:00.141177 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 24355
Jun 19 18:54:00 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:54:00.160089 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:00.142332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.15.91.3:45513)
0619 18:54:00.142969 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:00.142977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:00.143009 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:54:00.143073 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:00.143079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:00.143298 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:00.143488 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:00.144924 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:00.145946 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:00.147499 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:00.148134 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:00.148303 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:00.153115 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:00.153133 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:00.153150 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:00.153197 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:00.156590 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:00.157542 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:00.157549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:00.157552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:00.157694 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:00.158068 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:00.158071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:00.158074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:00.158450 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:00.158779 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:00.159321 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 18:54:00.159741 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":287,"threads_started":1}
I20260619 18:54:00.161582 24478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:00.246665 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 24355
I20260619 18:54:00.263846 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 23994
I20260619 18:54:00.265255 24056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:00.374188 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 23994
I20260619 18:54:00.396688 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:44637
--webserver_interface=127.15.91.62
--webserver_port=46451
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:44637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:54:00.756014 24501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:00.756448 24501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:00.756611 24501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:00.764497 24501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:54:00.764704 24501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:00.764825 24501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:00.764931 24501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:54:00.765017 24501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:54:00.776041 24501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:44637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:44637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=46451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:00.777612 24501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:00.779171 24501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:00.792798 24509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:00.792865 24507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:00.792819 24506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:00.793823 24501 server_base.cc:1034] running on GCE node
I20260619 18:54:00.794853 24501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:00.796674 24501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:00.797986 24501 hybrid_clock.cc:648] HybridClock initialized: now 1781895240797887 us; error 100 us; skew 500 ppm
I20260619 18:54:00.798534 24501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:54:00 dist-test-slave-dm3f krb5kdc[23981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895240, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:54:00.802976 24501 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:54:00.804883 24501 webserver.cc:460] Webserver started at http://127.15.91.62:46451/ using document root <none> and password file <none>
I20260619 18:54:00.805750 24501 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:00.805948 24501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:00.814699 24501 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:54:00.818506 24516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:00.820278 24501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:54:00.820567 24501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c"
format_stamp: "Formatted at 2026-06-19 18:53:57 on dist-test-slave-dm3f"
server_key: "fa42f25b0e0a91554839b28a91716ff3"
server_key_iv: "c3c5958a38ba4a7c8d61b68ce348da60"
server_key_version: "encryptionkey@0"
I20260619 18:54:00.821775 24501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:00.839812 24501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:00.845090 24501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:00.845434 24501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:00.869166 24501 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:44637
I20260619 18:54:00.869206 24567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:44637 every 8 connection(s)
I20260619 18:54:00.871099 24501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:54:00.871824 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 24501
I20260619 18:54:00.872951 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:38191
--local_ip_for_outbound_sockets=127.15.91.1
--tserver_master_addrs=127.15.91.62:44637
--webserver_port=36951
--webserver_interface=127.15.91.1
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.91.1
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:54:00.879552 24568 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:54:00.884500 24568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c: Bootstrap starting.
I20260619 18:54:00.909727 24568 log.cc:824] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:54:00.920624 24568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:54:00.921218 24568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c: Bootstrap complete.
I20260619 18:54:00.931156 24568 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } }
I20260619 18:54:00.932154 24568 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d5e74531dd419aa6ddfc9bd4f32a3c, State: Initialized, Role: FOLLOWER
I20260619 18:54:00.932847 24568 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } }
I20260619 18:54:00.933120 24568 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:54:00.933254 24568 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:54:00.933445 24568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:54:00.937656 24568 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } }
I20260619 18:54:00.938259 24568 leader_election.cc:302] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9d5e74531dd419aa6ddfc9bd4f32a3c; no voters:
I20260619 18:54:00.939015 24568 leader_election.cc:288] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:54:00.939182 24572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:54:00.940235 24572 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [term 2 LEADER]: Becoming Leader. State: Replica: a9d5e74531dd419aa6ddfc9bd4f32a3c, State: Running, Role: LEADER
I20260619 18:54:00.940874 24572 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } }
I20260619 18:54:00.943058 24568 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:54:00.943562 24573 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9d5e74531dd419aa6ddfc9bd4f32a3c. Latest consensus state: current_term: 2 leader_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } } }
I20260619 18:54:00.943964 24573 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [sys.catalog]: This master's current role is: LEADER
I20260619 18:54:00.944599 24574 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5e74531dd419aa6ddfc9bd4f32a3c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44637 } } }
I20260619 18:54:00.945005 24574 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c [sys.catalog]: This master's current role is: LEADER
I20260619 18:54:00.947324 24576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:54:00.955063 24576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:54:00.957198 24576 catalog_manager.cc:1259] Loaded cluster ID: 5b3ec0bff7eb4447bbebb1f30177315d
I20260619 18:54:00.957443 24576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:54:00.963677 24576 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:54:00.965895 24576 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9d5e74531dd419aa6ddfc9bd4f32a3c: Loaded TSK: 0
I20260619 18:54:00.968344 24576 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:54:01.287410 24570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:01.287806 24570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:01.287987 24570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:01.296590 24570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:01.296784 24570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:01.296983 24570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:54:01.308300 24570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.1.keytab
--principal=oryx/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:38191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=36951
--tserver_master_addrs=127.15.91.62:44637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:01.309930 24570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:01.311573 24570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:01.328047 24597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:01.328030 24595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:01.331351 24594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:01.336886 24570 server_base.cc:1034] running on GCE node
I20260619 18:54:01.337730 24570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:01.339076 24570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:01.340482 24570 hybrid_clock.cc:648] HybridClock initialized: now 1781895241340395 us; error 53 us; skew 500 ppm
I20260619 18:54:01.341167 24570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895241, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:54:01.349789 24570 init.cc:377] Logged in from keytab as oryx/127.15.91.1@KRBTEST.COM (short username oryx)
I20260619 18:54:01.351830 24570 webserver.cc:460] Webserver started at http://127.15.91.1:36951/ using document root <none> and password file <none>
I20260619 18:54:01.352741 24570 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:01.352912 24570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:01.359552 24570 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:54:01.362967 24604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:01.364675 24570 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:54:01.364912 24570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "c503a0f785684abfb40d7a2a53c13bee"
format_stamp: "Formatted at 2026-06-19 18:53:58 on dist-test-slave-dm3f"
server_key: "931887295d3a3ea3e26d58163b8bbdec"
server_key_iv: "88a12cf40aa24043679547fda9a033a3"
server_key_version: "encryptionkey@0"
I20260619 18:54:01.366098 24570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:01.379009 24570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:01.384171 24570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:01.384570 24570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:01.386190 24570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:01.388667 24570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:01.388806 24570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:01.388937 24570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:01.389021 24570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:01.430465 24570 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:38191
I20260619 18:54:01.430603 24716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:38191 every 8 connection(s)
I20260619 18:54:01.432179 24570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:54:01.442850 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 24570
I20260619 18:54:01.443497 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:38217
--local_ip_for_outbound_sockets=127.15.91.2
--tserver_master_addrs=127.15.91.62:44637
--webserver_port=37413
--webserver_interface=127.15.91.2
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.91.2
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:54:01.457011 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:01.435959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:49495 (local address 127.15.91.62:44637)
0619 18:54:01.436964 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:01.436985 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:01.437259 (+ 274us) server_negotiation.cc:407] Connection header received
0619 18:54:01.439788 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:01.439802 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:01.440208 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:01.440622 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:01.442410 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.444169 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.447466 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.448358 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.448601 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:01.455954 (+ 7353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:49495: BlockingRecv error: recv got EOF from 127.15.91.1:49495 (error 108)
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":428,"threads_started":1}
I20260619 18:54:01.457056 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:01.435985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:49495)
0619 18:54:01.437048 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:01.437105 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:01.439077 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:01.440802 (+ 1725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:01.440815 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:01.441503 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:01.442085 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.442106 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.444464 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.444474 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.447172 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.447203 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.448722 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.448733 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.449117 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:01.450014 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:01.450053 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:01.455565 (+ 5512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":442,"threads_started":1}
W20260619 18:54:01.457955 24717 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:44637 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.91.62:44637: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:01.468858 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:01.458748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:59441)
0619 18:54:01.458962 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:01.458985 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:01.459108 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:01.459526 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:01.459532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:01.459973 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:01.460135 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.460145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.461827 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.461832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.462958 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.462979 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.463801 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.463808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.464002 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:01.464712 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:01.464730 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:01.468533 (+ 3803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 18:54:01.469238 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:01.458912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:59441 (local address 127.15.91.62:44637)
0619 18:54:01.459133 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:01.459142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:01.459167 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:54:01.459232 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:01.459240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:01.459329 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:01.459445 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:01.460325 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.461598 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.463172 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.463610 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.463772 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:01.468785 (+ 5013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:59441: BlockingRecv error: recv got EOF from 127.15.91.1:59441 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:01.478680 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:01.469936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:58357)
0619 18:54:01.470130 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:01.470153 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:01.470280 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:01.470676 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:01.470681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:01.471094 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:01.471327 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.471337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.472515 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.472522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.473477 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.473491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.474281 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.474287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.474489 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:01.475234 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:01.475252 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:01.478318 (+ 3066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 18:54:01.478883 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:01.470052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:58357 (local address 127.15.91.62:44637)
0619 18:54:01.470273 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:01.470280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:01.470301 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:54:01.470414 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:01.470420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:01.470519 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:01.470659 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:01.471498 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.472327 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.473660 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.474102 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.474255 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:01.478579 (+ 4324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:58357: BlockingRecv error: recv got EOF from 127.15.91.1:58357 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 18:54:01.479117 24717 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:54:01.816849 24721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:01.817184 24721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:01.817348 24721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:01.827001 24721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:01.827142 24721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:01.827371 24721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:54:01.838380 24721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.2.keytab
--principal=oryx/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:38217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=37413
--tserver_master_addrs=127.15.91.62:44637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:01.840312 24721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:01.842594 24721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:01.860630 24726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:01.860661 24727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:01.860908 24729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:01.861968 24721 server_base.cc:1034] running on GCE node
I20260619 18:54:01.862926 24721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:01.864135 24721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:01.865516 24721 hybrid_clock.cc:648] HybridClock initialized: now 1781895241865414 us; error 114 us; skew 500 ppm
I20260619 18:54:01.866266 24721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895241, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:54:01.871367 24721 init.cc:377] Logged in from keytab as oryx/127.15.91.2@KRBTEST.COM (short username oryx)
I20260619 18:54:01.873590 24721 webserver.cc:460] Webserver started at http://127.15.91.2:37413/ using document root <none> and password file <none>
I20260619 18:54:01.874470 24721 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:01.874682 24721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:01.882195 24721 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20260619 18:54:01.885885 24736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:01.887778 24721 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260619 18:54:01.888056 24721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "adc328dc575044c99e33f1ab3ce7970a"
format_stamp: "Formatted at 2026-06-19 18:53:59 on dist-test-slave-dm3f"
server_key: "cef32c5f79c8dc7f21b517645f0629c4"
server_key_iv: "c262f9e47b9350f2de062905f495ba0f"
server_key_version: "encryptionkey@0"
I20260619 18:54:01.889220 24721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:01.912302 24721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:01.917943 24721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:01.918258 24721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:01.919430 24721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:01.920989 24721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:01.921141 24721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:01.921281 24721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:01.921370 24721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:01.959021 24721 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:38217
I20260619 18:54:01.959088 24848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:38217 every 8 connection(s)
I20260619 18:54:01.960649 24721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:54:01.967383 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 24721
I20260619 18:54:01.968145 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:45513
--local_ip_for_outbound_sockets=127.15.91.3
--tserver_master_addrs=127.15.91.62:44637
--webserver_port=38155
--webserver_interface=127.15.91.3
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.91.3
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:54:01.981645 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:01.963550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:44195 (local address 127.15.91.62:44637)
0619 18:54:01.963789 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:01.963798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:01.965232 (+ 1434us) server_negotiation.cc:407] Connection header received
0619 18:54:01.967790 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:01.967797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:01.967926 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:01.968095 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:01.970921 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.971916 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.975015 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.975638 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.975828 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:01.981365 (+ 5537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:44195: BlockingRecv error: recv got EOF from 127.15.91.2:44195 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 18:54:01.982479 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:01.964157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:44195)
0619 18:54:01.965011 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:01.965058 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:01.967115 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:01.968533 (+ 1418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:01.968550 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:01.969499 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:01.970526 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.970563 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.972181 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.972190 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.974690 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.974763 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.975976 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.975983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.976249 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:01.977071 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:01.977101 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:01.981143 (+ 4042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":324,"threads_started":1}
W20260619 18:54:01.983382 24849 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:44637 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.91.62:44637: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:01.993711 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:01.984135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:54101)
0619 18:54:01.984351 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:01.984383 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:01.984491 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:01.984901 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:01.984905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:01.985260 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:01.985429 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.985439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.987202 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.987207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.988197 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.988217 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.989075 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.989081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.989277 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:01.989962 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:01.989982 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:01.993373 (+ 3391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 18:54:01.993937 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:01.984271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:54101 (local address 127.15.91.62:44637)
0619 18:54:01.984490 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:01.984499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:01.984526 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:54:01.984596 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:01.984605 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:01.984707 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:01.984833 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:01.985613 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.986998 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.988395 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.988882 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.989049 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:01.993692 (+ 4643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:54101: BlockingRecv error: recv got EOF from 127.15.91.2:54101 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:01 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:02.003543 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:01.994786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:36951)
0619 18:54:01.994993 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:01.995038 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:01.995164 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:01.995532 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:01.995536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:01.995896 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:01.996059 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.996068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.997152 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.997159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.998106 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:01.998119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.998979 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.998986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:01.999195 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:01.999883 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:01.999902 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:02.003189 (+ 3287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 18:54:02.003680 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:01.994958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:36951 (local address 127.15.91.62:44637)
0619 18:54:01.995149 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:01.995155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:01.995177 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:54:01.995282 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:01.995288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:01.995386 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:01.995500 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:01.996228 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.996989 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.998275 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:01.998808 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:01.998948 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:02.003444 (+ 4496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:36951: BlockingRecv error: recv got EOF from 127.15.91.2:36951 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 18:54:02.003897 24849 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:54:02.352782 24852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:02.353325 24852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:02.353700 24852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:02.364305 24852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:02.364562 24852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:02.364795 24852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:54:02.377701 24852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/krb5kdc/oryx_127.15.91.3.keytab
--principal=oryx/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:45513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=38155
--tserver_master_addrs=127.15.91.62:44637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:02.379357 24852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:02.380903 24852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:02.397082 24858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:02.397341 24857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:02.397166 24860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:02.398413 24852 server_base.cc:1034] running on GCE node
I20260619 18:54:02.399632 24852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:02.400911 24852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:02.402209 24852 hybrid_clock.cc:648] HybridClock initialized: now 1781895242402148 us; error 49 us; skew 500 ppm
I20260619 18:54:02.402663 24852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:54:02 dist-test-slave-dm3f krb5kdc[23981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895242, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:54:02.406875 24852 init.cc:377] Logged in from keytab as oryx/127.15.91.3@KRBTEST.COM (short username oryx)
I20260619 18:54:02.408742 24852 webserver.cc:460] Webserver started at http://127.15.91.3:38155/ using document root <none> and password file <none>
I20260619 18:54:02.409621 24852 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:02.409823 24852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:02.416496 24852 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:54:02.420094 24867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:02.422752 24852 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:54:02.423133 24852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "acd14c7bedd049fea95200cf9b676d67"
format_stamp: "Formatted at 2026-06-19 18:53:59 on dist-test-slave-dm3f"
server_key: "592189c2908b8c4ce273f3bb8ed11652"
server_key_iv: "52a5b7a3b1d6dcb4883b1a4858313334"
server_key_version: "encryptionkey@0"
I20260619 18:54:02.424808 24852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:02.449892 24852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:02.455060 24852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:02.455368 24852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:02.456538 24852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:02.458168 24852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:02.458326 24852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:02.458477 24852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:02.458568 24852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:54:02 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:02 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:02.492010 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:02.479997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:44565)
0619 18:54:02.480921 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:02.480946 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:02.481141 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:02.481530 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:02.481535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:02.482233 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:02.482436 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:02.482447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.484376 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.484384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:02.485401 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:02.485418 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.486844 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.486853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:02.487125 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:02.487964 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:02.487994 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:02.491559 (+ 3565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":564,"threads_started":1}
W20260619 18:54:02.492424 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:02.480222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:44565 (local address 127.15.91.62:44637)
0619 18:54:02.480478 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:02.480487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:02.481063 (+ 576us) server_negotiation.cc:407] Connection header received
0619 18:54:02.481294 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:02.481300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:02.481377 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:02.481492 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:02.482753 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.484124 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.485624 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.486525 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.486707 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:02.491992 (+ 5285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:44565: BlockingRecv error: recv got EOF from 127.15.91.1:44565 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 18:54:02.499505 24852 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:45513
I20260619 18:54:02.499571 24980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:45513 every 8 connection(s)
I20260619 18:54:02.501173 24852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:54:02.508132 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 24852
Jun 19 18:54:02 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:02 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:54:02.522894 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:02.504312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:35701 (local address 127.15.91.62:44637)
0619 18:54:02.504531 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:02.504538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:02.505983 (+ 1445us) server_negotiation.cc:407] Connection header received
0619 18:54:02.508330 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:02.508335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:02.508423 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:02.508525 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:02.510929 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.512189 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.514111 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.514826 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.514984 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:02.522493 (+ 7509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:35701: BlockingRecv error: recv got EOF from 127.15.91.3:35701 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:54:02.524117 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:02.504930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:35701)
0619 18:54:02.505772 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:02.505818 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:02.507863 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:02.508885 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:02.508896 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:02.509898 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:02.510660 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:02.510682 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.512457 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.512462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:02.513927 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:02.513946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.515321 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.515330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:02.515735 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:02.516860 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:02.516904 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:02.522135 (+ 5231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":340,"threads_started":1}
W20260619 18:54:02.525022 24981 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:44637 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.91.62:44637: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
I20260619 18:54:02.529323 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:02.509234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.15.91.62:44637)
0619 18:54:02.510216 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:02.510221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:02.510236 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:54:02.510283 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:02.510287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:02.510345 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:02.510469 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:02.512684 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.513519 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.515373 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.515827 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.515996 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:02.519051 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:02.519065 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:02.519095 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:02.519163 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:02.524064 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:02.525968 (+ 1904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:02.525980 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:02.525996 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:02.526342 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:02.526956 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:02.526962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:02.526966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:02.527604 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:02.527998 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:02.528498 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 18:54:02.528944 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":674,"threads_started":1}
Jun 19 18:54:02 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:02 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:02.535995 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:02.525702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:50641)
0619 18:54:02.525952 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:02.525976 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:02.526100 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:02.526524 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:02.526529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:02.526978 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:02.527192 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:02.527206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.528402 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.528408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:02.529951 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:02.529971 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.530915 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.530920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:02.531064 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:02.531755 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:02.531776 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:02.535638 (+ 3862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260619 18:54:02.536183 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:02.525898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:50641 (local address 127.15.91.62:44637)
0619 18:54:02.526217 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:02.526226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:02.526246 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:54:02.526303 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:02.526307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:02.526368 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:02.526486 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:02.527366 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.528227 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.530131 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.530700 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.530888 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:02.535892 (+ 5004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:50641: BlockingRecv error: recv got EOF from 127.15.91.3:50641 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Jun 19 18:54:02 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:02 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:02.549176 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:02.537252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:57443)
0619 18:54:02.537510 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:02.537532 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:02.537641 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:02.537963 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:02.537967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:02.538305 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:02.538518 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:02.538527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.540386 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.540391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:02.541383 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:02.541402 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.542490 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.542497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:02.542709 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:02.543695 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:02.543721 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:02.548805 (+ 5084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260619 18:54:02.549292 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:02.537397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:57443 (local address 127.15.91.62:44637)
0619 18:54:02.537597 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:02.537601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:02.537622 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:54:02.537756 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:02.537760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:02.537828 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:02.537916 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:02.538783 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.539990 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.541667 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:02.542187 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:02.542329 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:02.549062 (+ 6733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:57443: BlockingRecv error: recv got EOF from 127.15.91.3:57443 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260619 18:54:02.549616 24981 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 18:54:03 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:03 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:03.015463 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:03.004638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:35029)
0619 18:54:03.005269 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:03.005290 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:03.005514 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:03.005907 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:03.005912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:03.006582 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:03.006903 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:03.006916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.008257 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.008264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:03.009289 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:03.009304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.010155 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.010163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:03.010370 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:03.011156 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:03.011177 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:03.015142 (+ 3965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":324,"threads_started":1}
W20260619 18:54:03.015746 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:03.004936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:35029 (local address 127.15.91.62:44637)
0619 18:54:03.005251 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:03.005266 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:03.005426 (+ 160us) server_negotiation.cc:407] Connection header received
0619 18:54:03.005685 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:03.005690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:03.005758 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:03.005883 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:03.007084 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.008080 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.009476 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.009940 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.010088 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:03.015437 (+ 5349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:35029: BlockingRecv error: recv got EOF from 127.15.91.2:35029 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Jun 19 18:54:03 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:03 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:03.505651 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:03.493823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:39205)
0619 18:54:03.494436 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:03.494457 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:03.494623 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:03.495251 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:03.495255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:03.495683 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:03.495856 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:03.495866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.497448 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.497453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:03.498667 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:03.498690 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.499707 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.499715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:03.499926 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:03.500685 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:03.500711 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:03.505296 (+ 4585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":333,"threads_started":1}
W20260619 18:54:03.505931 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:03.493948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:39205 (local address 127.15.91.62:44637)
0619 18:54:03.494142 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:03.494148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:03.494700 (+ 552us) server_negotiation.cc:407] Connection header received
0619 18:54:03.494961 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:03.494966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:03.495041 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:03.495143 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:03.496120 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.497241 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.498897 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.499482 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.499600 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:03.505657 (+ 6057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:39205: BlockingRecv error: recv got EOF from 127.15.91.1:39205 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 18:54:03 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:03 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:03.560988 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:03.550410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:51199)
0619 18:54:03.551104 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:03.551127 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:03.551287 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:03.551690 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:03.551695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:03.552213 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:03.552410 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:03.552419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.553539 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.553546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:03.554585 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:03.554601 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.555526 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.555532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:03.555713 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:03.556561 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:03.556593 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:03.560605 (+ 4012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":384,"threads_started":1}
W20260619 18:54:03.561167 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:03.550935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:51199 (local address 127.15.91.62:44637)
0619 18:54:03.551178 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:03.551186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:03.551209 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:54:03.551433 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:03.551440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:03.551537 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:03.551662 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:03.552596 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.553376 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.554835 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:03.555287 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:03.555444 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:03.560869 (+ 5425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:51199: BlockingRecv error: recv got EOF from 127.15.91.3:51199 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 18:54:04 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:04 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:04.029608 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:04.017079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:44129)
0619 18:54:04.017888 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:04.017907 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:04.018103 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:04.018694 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:04.018706 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:04.019343 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:04.019742 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:04.019770 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.021458 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.021464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:04.022695 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:04.022766 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.023941 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.023947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:04.024091 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:04.024819 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:04.024841 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:04.029311 (+ 4470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":554,"threads_started":1}
W20260619 18:54:04.030104 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:04.017190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:44129 (local address 127.15.91.62:44637)
0619 18:54:04.017404 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:04.017411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:04.018041 (+ 630us) server_negotiation.cc:407] Connection header received
0619 18:54:04.018260 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:04.018265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:04.018348 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:04.018467 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:04.020038 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.021153 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.023085 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.023619 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.023771 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:04.029822 (+ 6051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:44129: BlockingRecv error: recv got EOF from 127.15.91.2:44129 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 18:54:04 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:04 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:04.516207 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:04.506995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:54111)
0619 18:54:04.507550 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:04.507570 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:04.507697 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:04.508078 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:04.508083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:04.508471 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:04.508649 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:04.508659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.509752 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.509757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:04.510700 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:04.510735 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.511539 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.511546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:04.511740 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:04.512424 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:04.512442 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:04.515794 (+ 3352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":311,"threads_started":1}
W20260619 18:54:04.516383 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:04.507138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:54111 (local address 127.15.91.62:44637)
0619 18:54:04.507338 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:04.507343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:04.507662 (+ 319us) server_negotiation.cc:407] Connection header received
0619 18:54:04.507847 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:04.507853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:04.507929 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:04.508039 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:04.508827 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.509580 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.510896 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.511363 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.511513 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:04.516073 (+ 4560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:54111: BlockingRecv error: recv got EOF from 127.15.91.1:54111 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 18:54:04 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:04 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:04.573685 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:04.562651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:50021)
0619 18:54:04.563609 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:04.563633 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:04.563802 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:04.564250 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:04.564254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:04.564747 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:04.564914 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:04.564924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.566105 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.566111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:04.567222 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:04.567241 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.568089 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.568094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:04.568221 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:04.569098 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:04.569128 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:04.573291 (+ 4163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":557,"threads_started":1}
W20260619 18:54:04.573812 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:04.562899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:50021 (local address 127.15.91.62:44637)
0619 18:54:04.563144 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:04.563150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:04.563818 (+ 668us) server_negotiation.cc:407] Connection header received
0619 18:54:04.563910 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:04.563914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:04.564000 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:04.564221 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:04.565133 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.565910 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.567431 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:04.567882 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:04.568004 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:04.573508 (+ 5504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:50021: BlockingRecv error: recv got EOF from 127.15.91.3:50021 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 18:54:05 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:05 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:05.043308 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:05.030937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:56701)
0619 18:54:05.031852 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:05.031872 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:05.032056 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:05.032607 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:05.032614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:05.033229 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:05.033485 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:05.033499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.035202 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.035209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:05.036339 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:05.036361 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.037603 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.037610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:05.037825 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:05.038804 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:05.038842 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:05.042936 (+ 4094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":445,"threads_started":1}
W20260619 18:54:05.043555 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:05.031285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:56701 (local address 127.15.91.62:44637)
0619 18:54:05.031542 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:05.031549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:05.031979 (+ 430us) server_negotiation.cc:407] Connection header received
0619 18:54:05.032220 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:05.032228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:05.032326 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:05.032453 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:05.033703 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.034960 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.036638 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.037251 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.037398 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:05.043304 (+ 5906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:56701: BlockingRecv error: recv got EOF from 127.15.91.2:56701 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Jun 19 18:54:05 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:05 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:05.532014 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:05.517696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:56135)
0619 18:54:05.518414 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:05.518435 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:05.518600 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:05.519255 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:05.519261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:05.519723 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:05.519948 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:05.519960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.521652 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.521658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:05.523832 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:05.523863 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.525112 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.525118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:05.525412 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:05.526631 (+ 1219us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:05.526664 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:05.531714 (+ 5050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":420,"threads_started":1}
W20260619 18:54:05.532331 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:05.517891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:56135 (local address 127.15.91.62:44637)
0619 18:54:05.518119 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:05.518123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:05.518613 (+ 490us) server_negotiation.cc:407] Connection header received
0619 18:54:05.518854 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:05.518859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:05.518950 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:05.519052 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:05.520251 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.521394 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.524210 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.524827 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.525014 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:05.532062 (+ 7048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:56135: BlockingRecv error: recv got EOF from 127.15.91.1:56135 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 19 18:54:05 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:05 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:05.587569 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:05.575184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:41009)
0619 18:54:05.575821 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:05.575848 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:05.576015 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:05.576683 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:05.576706 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:05.577198 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:05.577467 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:05.577481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.579241 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.579248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:05.580578 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:05.580607 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.581830 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.581837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:05.582042 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:05.583020 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:05.583050 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:05.587228 (+ 4178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":373,"threads_started":1}
W20260619 18:54:05.588184 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:05.575512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:41009 (local address 127.15.91.62:44637)
0619 18:54:05.575804 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:05.575809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:05.575969 (+ 160us) server_negotiation.cc:407] Connection header received
0619 18:54:05.576227 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:05.576233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:05.576340 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:05.576480 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:05.577753 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.578919 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.580938 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:05.581523 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:05.581690 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:05.587817 (+ 6127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:41009: BlockingRecv error: recv got EOF from 127.15.91.3:41009 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 18:54:06 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:06 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:06.056442 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:06.044621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:52043)
0619 18:54:06.045246 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:06.045265 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:06.045397 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:06.045977 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:06.045980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:06.046367 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:06.046567 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:06.046577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.048459 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.048466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:06.050554 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:06.050576 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.052016 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.052022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:06.052185 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:06.052999 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:06.053022 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:06.056155 (+ 3133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":369,"threads_started":1}
W20260619 18:54:06.056947 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:06.044750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:52043 (local address 127.15.91.62:44637)
0619 18:54:06.045026 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:06.045037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:06.045385 (+ 348us) server_negotiation.cc:407] Connection header received
0619 18:54:06.045560 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:06.045570 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:06.045721 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:06.045873 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:06.046813 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.048264 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.050803 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.051748 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.051971 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:06.056606 (+ 4635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:52043: BlockingRecv error: recv got EOF from 127.15.91.2:52043 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 18:54:06 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:06 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:06.547083 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:06.533379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:60193)
0619 18:54:06.533981 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:06.533999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:06.534179 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:06.534923 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:06.534927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:06.535367 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:06.535581 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:06.535592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.536860 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.536865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:06.537892 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:06.537910 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.538834 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.538839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:06.538989 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:06.542860 (+ 3871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:06.542887 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:06.546674 (+ 3787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":302,"threads_started":1}
W20260619 18:54:06.547188 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:06.533449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:60193 (local address 127.15.91.62:44637)
0619 18:54:06.533631 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:06.533635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:06.534085 (+ 450us) server_negotiation.cc:407] Connection header received
0619 18:54:06.534351 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:06.534355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:06.534639 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:06.534845 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:06.535838 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.536663 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.538107 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.538563 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.538681 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:06.546936 (+ 8255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:60193: BlockingRecv error: recv got EOF from 127.15.91.1:60193 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 18:54:06 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:06 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:06.598448 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:06.588726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:41437)
0619 18:54:06.589301 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:06.589320 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:06.589467 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:06.589848 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:06.589852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:06.590273 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:06.590445 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:06.590456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.591595 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.591600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:06.592713 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:06.592734 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.593625 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.593629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:06.593864 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:06.594502 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:06.594523 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:06.598083 (+ 3560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":331,"threads_started":1}
W20260619 18:54:06.598536 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:06.588851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:41437 (local address 127.15.91.62:44637)
0619 18:54:06.589100 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:06.589104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:06.589409 (+ 305us) server_negotiation.cc:407] Connection header received
0619 18:54:06.589592 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:06.589596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:06.589662 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:06.589762 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:06.590633 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.591410 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.592904 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:06.593461 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:06.593591 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:06.598297 (+ 4706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:41437: BlockingRecv error: recv got EOF from 127.15.91.3:41437 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 18:54:07 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:07 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:07.069382 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:07.057895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:54401)
0619 18:54:07.059029 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:07.059054 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:07.059245 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:07.059799 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:07.059803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:07.060344 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:07.060619 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:07.060636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.062039 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.062045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:07.063133 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:07.063153 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.064412 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.064418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:07.064610 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:07.065470 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:07.065507 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:07.069112 (+ 3605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":749,"threads_started":1}
W20260619 18:54:07.069777 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:07.058217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:54401 (local address 127.15.91.62:44637)
0619 18:54:07.058600 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:07.058617 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:07.059317 (+ 700us) server_negotiation.cc:407] Connection header received
0619 18:54:07.059454 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:07.059463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:07.059570 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:07.059707 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:07.060814 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.061766 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.063394 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.064146 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.064290 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:07.069482 (+ 5192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:54401: BlockingRecv error: recv got EOF from 127.15.91.2:54401 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
Jun 19 18:54:07 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:07 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:07.559367 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:07.548269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:46467)
0619 18:54:07.548869 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:07.548898 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:07.549021 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:07.549328 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:07.549332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:07.549738 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:07.549911 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:07.549920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.551355 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.551362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:07.552650 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:07.552679 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.553913 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.553920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:07.554108 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:07.554980 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:07.555001 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:07.559080 (+ 4079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":356,"threads_started":1}
W20260619 18:54:07.559675 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:07.548291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:46467 (local address 127.15.91.62:44637)
0619 18:54:07.548469 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:07.548474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:07.548998 (+ 524us) server_negotiation.cc:407] Connection header received
0619 18:54:07.549111 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:07.549114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:07.549183 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:07.549282 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:07.550168 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.550991 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.552916 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.553602 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.553801 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:07.559403 (+ 5602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:46467: BlockingRecv error: recv got EOF from 127.15.91.1:46467 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 18:54:07 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:07 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:07.611052 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:07.600090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:43583)
0619 18:54:07.600697 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:07.600716 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:07.600915 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:07.601460 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:07.601468 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:07.602062 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:07.602274 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:07.602286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.603529 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.603536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:07.604516 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:07.604533 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.605511 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.605535 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:07.605969 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:07.607039 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:07.607063 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:07.610632 (+ 3569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":310,"threads_started":1}
W20260619 18:54:07.611166 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:07.600175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:43583 (local address 127.15.91.62:44637)
0619 18:54:07.600383 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:07.600392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:07.600836 (+ 444us) server_negotiation.cc:407] Connection header received
0619 18:54:07.601199 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:07.601204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:07.601275 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:07.601376 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:07.602443 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.603345 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.604715 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:07.605180 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:07.605339 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:07.610917 (+ 5578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:43583: BlockingRecv error: recv got EOF from 127.15.91.3:43583 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 18:54:08 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:08 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:08.080566 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:08.070507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:59679)
0619 18:54:08.071227 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:08.071253 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:08.071420 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:08.071811 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:08.071826 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:08.072312 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:08.072487 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:08.072497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.073785 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.073803 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:08.074941 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:08.074964 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.076000 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.076006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:08.076160 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:08.076962 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:08.076982 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:08.080316 (+ 3334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":429,"threads_started":1}
W20260619 18:54:08.080971 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:08.070629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:59679 (local address 127.15.91.62:44637)
0619 18:54:08.070947 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:08.070957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:08.071359 (+ 402us) server_negotiation.cc:407] Connection header received
0619 18:54:08.071542 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:08.071546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:08.071616 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:08.071767 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:08.072653 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.073437 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.075200 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.075742 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.075887 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:08.080642 (+ 4755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:59679: BlockingRecv error: recv got EOF from 127.15.91.2:59679 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 19 18:54:08 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:08 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:08.570374 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:08.560683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:43063)
0619 18:54:08.561320 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:08.561338 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:08.561556 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:08.561934 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:08.561940 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:08.562421 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:08.562605 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:08.562615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.563806 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.563819 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:08.564916 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:08.564931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.565859 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.565866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:08.566056 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:08.566820 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:08.566842 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:08.569948 (+ 3106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":310,"threads_started":1}
W20260619 18:54:08.570504 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:08.560806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:43063 (local address 127.15.91.62:44637)
0619 18:54:08.561096 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:08.561105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:08.561458 (+ 353us) server_negotiation.cc:407] Connection header received
0619 18:54:08.561709 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:08.561714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:08.561783 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:08.561901 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:08.562804 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.563624 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.565094 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.565670 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.565831 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:08.570211 (+ 4380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:43063: BlockingRecv error: recv got EOF from 127.15.91.1:43063 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 19 18:54:08 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:08 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:08.623171 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:08.612536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:56963)
0619 18:54:08.613353 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:08.613378 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:08.613545 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:08.613966 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:08.613970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:08.614391 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:08.614596 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:08.614606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.616267 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.616275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:08.617720 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:08.617749 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.618889 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.618896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:08.619084 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:08.619772 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:08.619793 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:08.622898 (+ 3105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":421,"threads_started":1}
W20260619 18:54:08.623634 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:08.612554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:56963 (local address 127.15.91.62:44637)
0619 18:54:08.612933 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:08.612944 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:08.613538 (+ 594us) server_negotiation.cc:407] Connection header received
0619 18:54:08.613671 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:08.613675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:08.613773 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:08.613874 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:08.614847 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.615971 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.617998 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:08.618564 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:08.618696 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:08.623174 (+ 4478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:56963: BlockingRecv error: recv got EOF from 127.15.91.3:56963 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Jun 19 18:54:09 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:09 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:09.091915 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:09.081786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:49105)
0619 18:54:09.082459 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:09.082477 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:09.082601 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:09.083013 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:09.083017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:09.083508 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:09.083680 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:09.083689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.085092 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.085100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:09.086123 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:09.086138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.087069 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.087076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:09.087255 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:09.087894 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:09.087923 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:09.091505 (+ 3582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":325,"threads_started":1}
W20260619 18:54:09.092026 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:09.081849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:49105 (local address 127.15.91.62:44637)
0619 18:54:09.082078 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:09.082083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:09.082575 (+ 492us) server_negotiation.cc:407] Connection header received
0619 18:54:09.082782 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:09.082790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:09.082865 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:09.082986 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:09.083852 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.084882 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.086315 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.086876 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.087021 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:09.091764 (+ 4743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:49105: BlockingRecv error: recv got EOF from 127.15.91.2:49105 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 18:54:09 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:09 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:09.580921 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:09.571502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:33809)
0619 18:54:09.572077 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:09.572096 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:09.572231 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:09.572572 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:09.572576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:09.573012 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:09.573175 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:09.573185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.574290 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.574295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:09.575313 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:09.575330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.576212 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.576218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:09.576381 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:09.577132 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:09.577151 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:09.580563 (+ 3412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":331,"threads_started":1}
W20260619 18:54:09.581004 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:09.571647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:33809 (local address 127.15.91.62:44637)
0619 18:54:09.571921 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:09.571927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:09.572199 (+ 272us) server_negotiation.cc:407] Connection header received
0619 18:54:09.572349 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:09.572354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:09.572426 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:09.572520 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:09.573345 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.574095 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.575528 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.575995 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.576110 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:09.580794 (+ 4684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:33809: BlockingRecv error: recv got EOF from 127.15.91.1:33809 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 19 18:54:09 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:09 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:09.634474 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:09.624274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:42471)
0619 18:54:09.624860 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:09.624879 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:09.625026 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:09.625472 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:09.625475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:09.625945 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:09.626133 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:09.626142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.627615 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.627623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:09.628834 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:09.628854 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.630070 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.630077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:09.630284 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:09.631060 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:09.631082 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:09.634191 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":264,"threads_started":1}
W20260619 18:54:09.634829 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:09.624543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:42471 (local address 127.15.91.62:44637)
0619 18:54:09.624797 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:09.624806 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:09.624979 (+ 173us) server_negotiation.cc:407] Connection header received
0619 18:54:09.625191 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:09.625196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:09.625286 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:09.625459 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:09.626290 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.627303 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.629137 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:09.629841 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:09.630008 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:09.634482 (+ 4474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:42471: BlockingRecv error: recv got EOF from 127.15.91.3:42471 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 19 18:54:10 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:10 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:10.104398 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:10.093287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:56461)
0619 18:54:10.093940 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:10.093956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:10.094147 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:10.094532 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:10.094536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:10.095058 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:10.095222 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:10.095230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.096451 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.096457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:10.097416 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:10.097430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.098302 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.098309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:10.098484 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:10.099987 (+ 1503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:10.100009 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:10.104032 (+ 4023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":381,"threads_started":1}
W20260619 18:54:10.104557 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:10.093394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:56461 (local address 127.15.91.62:44637)
0619 18:54:10.093712 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:10.093720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:10.094064 (+ 344us) server_negotiation.cc:407] Connection header received
0619 18:54:10.094291 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:10.094296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:10.094384 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:10.094500 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:10.095401 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.096244 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.097604 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.098067 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.098218 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:10.104293 (+ 6075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:56461: BlockingRecv error: recv got EOF from 127.15.91.2:56461 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
Jun 19 18:54:10 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:10 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:10.591758 25008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:10.582066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:43463)
0619 18:54:10.582968 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:10.582994 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:10.583165 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:10.583531 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:10.583536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:10.583997 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:10.584162 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:10.584171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.585285 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.585292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:10.586291 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:10.586306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.587231 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.587238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:10.587436 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:10.588112 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:10.588132 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:10.591455 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":560,"threads_started":1}
W20260619 18:54:10.592129 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:10.582130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:43463 (local address 127.15.91.62:44637)
0619 18:54:10.582714 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:10.582720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:10.583095 (+ 375us) server_negotiation.cc:407] Connection header received
0619 18:54:10.583315 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:10.583320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:10.583386 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:10.583496 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:10.584371 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.585101 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.586483 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.587046 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.587197 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:10.591817 (+ 4620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:43463: BlockingRecv error: recv got EOF from 127.15.91.1:43463 (error 108)
Metrics: {"server-negotiator.queue_time_us":211}
Jun 19 18:54:10 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:10 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:10.646651 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:10.635791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:35623)
0619 18:54:10.636371 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:10.636388 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:10.636524 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:10.637019 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:10.637025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:10.637638 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:10.637810 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:10.637821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.639401 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.639407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:10.640409 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:10.640427 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.641422 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.641428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:10.641582 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:10.642267 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:10.642288 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:10.646323 (+ 4035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":292,"threads_started":1}
W20260619 18:54:10.646965 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:10.635867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:35623 (local address 127.15.91.62:44637)
0619 18:54:10.636077 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:10.636081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:10.636490 (+ 409us) server_negotiation.cc:407] Connection header received
0619 18:54:10.636635 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:10.636639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:10.636733 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:10.636829 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:10.638065 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.639173 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.640642 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:10.641131 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:10.641268 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:10.646650 (+ 5382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:35623: BlockingRecv error: recv got EOF from 127.15.91.3:35623 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 18:54:11 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:11 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:11.115151 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:11.105575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:43333)
0619 18:54:11.106160 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:11.106177 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:11.106305 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:11.106641 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:11.106644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:11.107120 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:11.107292 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:11.107301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.108396 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.108400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:11.109388 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:11.109402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.110212 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.110217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:11.110349 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:11.111107 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:11.111126 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:11.114688 (+ 3562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":327,"threads_started":1}
W20260619 18:54:11.115341 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:11.105725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:43333 (local address 127.15.91.62:44637)
0619 18:54:11.105900 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:11.105905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:11.106287 (+ 382us) server_negotiation.cc:407] Connection header received
0619 18:54:11.106423 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:11.106428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:11.106498 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:11.106594 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:11.107474 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.108258 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.109596 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.110045 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.110156 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:11.115028 (+ 4872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:43333: BlockingRecv error: recv got EOF from 127.15.91.2:43333 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 18:54:11 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:11 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:11.608012 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:11.592856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:42635)
0619 18:54:11.593536 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:11.593554 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:11.593678 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:11.594044 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:11.594047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:11.594523 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:11.594678 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:11.594693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.596702 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.596709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:11.597828 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:11.597850 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.598957 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.598965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:11.599151 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:11.604318 (+ 5167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:11.604342 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:11.607630 (+ 3288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":434,"threads_started":1}
W20260619 18:54:11.608127 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:11.593000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:42635 (local address 127.15.91.62:44637)
0619 18:54:11.593222 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:11.593231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:11.593644 (+ 413us) server_negotiation.cc:407] Connection header received
0619 18:54:11.593787 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:11.593791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:11.593891 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:11.594002 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:11.595089 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.596410 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.598125 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.598613 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.598864 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:11.607874 (+ 9010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:42635: BlockingRecv error: recv got EOF from 127.15.91.1:42635 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 18:54:11 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:11 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:11.659031 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:11.647951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:34079)
0619 18:54:11.648445 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:11.648463 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:11.648644 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:11.649084 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:11.649089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:11.649582 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:11.649752 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:11.649762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.650959 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.650967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:11.651947 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:11.651961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.652812 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.652819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:11.653007 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:11.653690 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:11.653711 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:11.658594 (+ 4883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":245,"threads_started":1}
W20260619 18:54:11.659135 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:11.647955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:34079 (local address 127.15.91.62:44637)
0619 18:54:11.648170 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:11.648178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:11.648572 (+ 394us) server_negotiation.cc:407] Connection header received
0619 18:54:11.648785 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:11.648793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:11.648942 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:11.649047 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:11.649942 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.650793 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.652144 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:11.652615 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:11.652763 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:11.658907 (+ 6144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:34079: BlockingRecv error: recv got EOF from 127.15.91.3:34079 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 18:54:12 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:12 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:12.127728 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:12.116590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:43897)
0619 18:54:12.117194 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:12.117215 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:12.117359 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:12.117984 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:12.117989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:12.118654 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:12.119099 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:12.119110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.120693 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.120698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:12.121724 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:12.121741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.122896 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.122903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:12.123102 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:12.123907 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:12.123928 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:12.127461 (+ 3533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":363,"threads_started":1}
W20260619 18:54:12.128036 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:12.116815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:43897 (local address 127.15.91.62:44637)
0619 18:54:12.117018 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:12.117023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:12.117390 (+ 367us) server_negotiation.cc:407] Connection header received
0619 18:54:12.117495 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:12.117501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:12.117707 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:12.117846 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:12.119426 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.120479 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.121933 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.122630 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.122816 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:12.127727 (+ 4911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:43897: BlockingRecv error: recv got EOF from 127.15.91.2:43897 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 18:54:12 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:12 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:12.619329 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:12.609214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:57731)
0619 18:54:12.609779 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:12.609798 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:12.609919 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:12.610278 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:12.610283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:12.610843 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:12.611037 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:12.611047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.612303 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.612310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:12.613579 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:12.613605 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.614503 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.614508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:12.614636 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:12.615465 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:12.615491 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:12.618911 (+ 3420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":341,"threads_started":1}
W20260619 18:54:12.619496 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:12.609350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:57731 (local address 127.15.91.62:44637)
0619 18:54:12.609651 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:12.609658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:12.609884 (+ 226us) server_negotiation.cc:407] Connection header received
0619 18:54:12.610020 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:12.610023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:12.610106 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:12.610194 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:12.611247 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.612024 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.613816 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.614288 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.614430 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:12.619182 (+ 4752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:57731: BlockingRecv error: recv got EOF from 127.15.91.1:57731 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
Jun 19 18:54:12 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:12 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:12.669202 25015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:12.660187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:38401)
0619 18:54:12.660796 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:12.660813 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:12.660988 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:12.661355 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:12.661359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:12.661795 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:12.661947 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:12.661956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.663111 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.663122 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:12.664073 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:12.664086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.664911 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.664918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:12.665105 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:12.665820 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:12.665840 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:12.668942 (+ 3102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":408,"threads_started":1}
W20260619 18:54:12.669632 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:12.660267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:38401 (local address 127.15.91.62:44637)
0619 18:54:12.660462 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:12.660468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:12.660920 (+ 452us) server_negotiation.cc:407] Connection header received
0619 18:54:12.661119 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:12.661124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:12.661215 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:12.661326 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:12.662118 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.662937 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.664255 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:12.664722 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:12.664880 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:12.669199 (+ 4319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:38401: BlockingRecv error: recv got EOF from 127.15.91.3:38401 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 18:54:13 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:13 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:13.138298 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:13.128857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:45141)
0619 18:54:13.129413 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:13.129430 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:13.129604 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:13.130017 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:13.130022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:13.130628 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:13.130935 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:13.130953 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.131999 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.132005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:13.132936 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:13.132950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.133761 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.133767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:13.133958 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:13.134676 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:13.134695 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:13.137912 (+ 3217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":294,"threads_started":1}
W20260619 18:54:13.138401 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:13.129166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:45141 (local address 127.15.91.62:44637)
0619 18:54:13.129370 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:13.129377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:13.129548 (+ 171us) server_negotiation.cc:407] Connection header received
0619 18:54:13.129735 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:13.129744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:13.129857 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:13.129981 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:13.131089 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.131828 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.133139 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.133583 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.133730 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:13.138154 (+ 4424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:45141: BlockingRecv error: recv got EOF from 127.15.91.2:45141 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 18:54:13 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:13 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:13.630589 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:13.620731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:54383)
0619 18:54:13.621299 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:13.621316 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:13.621505 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:13.621904 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:13.621909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:13.622336 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:13.622566 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:13.622577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.623718 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.623724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:13.624687 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:13.624702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.625533 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.625539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:13.625730 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:13.626480 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:13.626501 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:13.630206 (+ 3705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":316,"threads_started":1}
W20260619 18:54:13.630759 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:13.620829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:54383 (local address 127.15.91.62:44637)
0619 18:54:13.621090 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:13.621099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:13.621421 (+ 322us) server_negotiation.cc:407] Connection header received
0619 18:54:13.621641 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:13.621647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:13.621723 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:13.621838 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:13.622755 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.623543 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.624875 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.625339 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.625504 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:13.630501 (+ 4997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:54383: BlockingRecv error: recv got EOF from 127.15.91.1:54383 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 19 18:54:13 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:13 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:13.679636 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:13.670319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:52303)
0619 18:54:13.671061 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:13.671078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:13.671247 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:13.671666 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:13.671670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:13.672121 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:13.672300 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:13.672311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.673378 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.673384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:13.674331 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:13.674345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.675255 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.675262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:13.675436 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:13.676070 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:13.676089 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:13.679256 (+ 3167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":458,"threads_started":1}
W20260619 18:54:13.679805 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:13.670415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:52303 (local address 127.15.91.62:44637)
0619 18:54:13.670633 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:13.670639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:13.671182 (+ 543us) server_negotiation.cc:407] Connection header received
0619 18:54:13.671409 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:13.671422 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:13.671509 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:13.671631 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:13.672491 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.673211 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.674515 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:13.675058 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:13.675218 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:13.679523 (+ 4305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:52303: BlockingRecv error: recv got EOF from 127.15.91.3:52303 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 18:54:14 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:14 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:14.149930 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:14.139629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:43509)
0619 18:54:14.140379 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:14.140400 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:14.140759 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:14.141232 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:14.141237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:14.141721 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:14.141899 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:14.141908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.143106 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.143113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:14.144159 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:14.144175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.145065 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.145071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:14.145273 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:14.146166 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:14.146187 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:14.149550 (+ 3363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":411,"threads_started":1}
W20260619 18:54:14.150116 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:14.139702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:43509 (local address 127.15.91.62:44637)
0619 18:54:14.140122 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:14.140130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:14.140534 (+ 404us) server_negotiation.cc:407] Connection header received
0619 18:54:14.140982 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:14.140987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:14.141060 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:14.141178 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:14.142079 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.142933 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.144340 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.144885 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.145039 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:14.149831 (+ 4792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:43509: BlockingRecv error: recv got EOF from 127.15.91.2:43509 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
Jun 19 18:54:14 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:14 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:14.641379 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:14.631849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:40747)
0619 18:54:14.632536 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:14.632554 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:14.632741 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:14.633122 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:14.633127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:14.633592 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:14.633756 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:14.633766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.635048 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.635054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:14.636164 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:14.636181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.637089 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.637096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:14.637271 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:14.637902 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:14.637920 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:14.641007 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":367,"threads_started":1}
W20260619 18:54:14.641515 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:14.632032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:40747 (local address 127.15.91.62:44637)
0619 18:54:14.632328 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:14.632337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:14.632661 (+ 324us) server_negotiation.cc:407] Connection header received
0619 18:54:14.632895 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:14.632900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:14.632969 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:14.633088 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:14.633927 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.634856 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.636395 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.636875 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.637032 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:14.641259 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:40747: BlockingRecv error: recv got EOF from 127.15.91.1:40747 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Jun 19 18:54:14 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:14 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:14.690186 25021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:14.681032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:55119)
0619 18:54:14.681607 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:14.681625 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:14.681757 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:14.682108 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:14.682111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:14.682503 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:14.682662 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:14.682671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.683834 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.683839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:14.684969 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:14.684984 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.686028 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.686034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:14.686250 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:14.686981 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:14.686999 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:14.689936 (+ 2937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":337,"threads_started":1}
W20260619 18:54:14.690665 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:14.681145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:55119 (local address 127.15.91.62:44637)
0619 18:54:14.681402 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:14.681407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:14.681720 (+ 313us) server_negotiation.cc:407] Connection header received
0619 18:54:14.681868 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:14.681873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:14.681958 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:14.682061 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:14.682874 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.683639 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.685179 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:14.685832 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:14.685974 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:14.690207 (+ 4233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:55119: BlockingRecv error: recv got EOF from 127.15.91.3:55119 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 18:54:15 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:15 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:15.160668 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:15.151415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:36307)
0619 18:54:15.151949 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:15.151967 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:15.152094 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:15.152432 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:15.152435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:15.152855 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:15.153020 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:15.153030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.154215 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.154220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:15.155555 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:15.155573 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.156418 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.156423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:15.156575 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:15.157216 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:15.157235 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:15.160324 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":295,"threads_started":1}
W20260619 18:54:15.160902 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:15.151586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:36307 (local address 127.15.91.62:44637)
0619 18:54:15.151870 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:15.151877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:15.152058 (+ 181us) server_negotiation.cc:407] Connection header received
0619 18:54:15.152202 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:15.152206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:15.152293 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:15.152391 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:15.153211 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.154035 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.155735 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.156203 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.156337 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:15.160635 (+ 4298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:36307: BlockingRecv error: recv got EOF from 127.15.91.2:36307 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 19 18:54:15 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:15 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:15.652204 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:15.642454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:47923)
0619 18:54:15.643042 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:15.643059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:15.643215 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:15.643524 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:15.643527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:15.643906 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:15.644127 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:15.644140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.645309 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.645314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:15.646354 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:15.646369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.647733 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.647805 (+ 72us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:15.647998 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:15.648652 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:15.648670 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:15.651947 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":352,"threads_started":1}
W20260619 18:54:15.652917 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:15.642634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:47923 (local address 127.15.91.62:44637)
0619 18:54:15.642814 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:15.642819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:15.643180 (+ 361us) server_negotiation.cc:407] Connection header received
0619 18:54:15.643322 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:15.643327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:15.643393 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:15.643477 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:15.644344 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.645101 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.646864 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.647509 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.647630 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:15.652549 (+ 4919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:47923: BlockingRecv error: recv got EOF from 127.15.91.1:47923 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 18:54:15 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:15 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:15.700732 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:15.691375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:58657)
0619 18:54:15.691943 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:15.691961 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:15.692089 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:15.692594 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:15.692598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:15.693121 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:15.693286 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:15.693295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.694649 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.694654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:15.695665 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:15.695681 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.696551 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.696557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:15.696696 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:15.697379 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:15.697398 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:15.700445 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":309,"threads_started":1}
W20260619 18:54:15.701063 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:15.691485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:58657 (local address 127.15.91.62:44637)
0619 18:54:15.691753 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:15.691757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:15.692047 (+ 290us) server_negotiation.cc:407] Connection header received
0619 18:54:15.692259 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:15.692264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:15.692402 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:15.692534 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:15.693425 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.694433 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.695905 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:15.696351 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:15.696466 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:15.700744 (+ 4278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:58657: BlockingRecv error: recv got EOF from 127.15.91.3:58657 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Jun 19 18:54:16 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:16 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:16.171993 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:16.162106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:55911)
0619 18:54:16.162814 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:16.162833 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:16.163031 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:16.163426 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:16.163431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:16.163854 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:16.164037 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:16.164047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.165355 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.165362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:16.166454 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:16.166470 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.167389 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.167396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:16.167589 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:16.168215 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:16.168232 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:16.171713 (+ 3481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":363,"threads_started":1}
W20260619 18:54:16.172271 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:16.162317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:55911 (local address 127.15.91.62:44637)
0619 18:54:16.162605 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:16.162613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:16.162948 (+ 335us) server_negotiation.cc:407] Connection header received
0619 18:54:16.163191 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:16.163197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:16.163276 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:16.163389 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:16.164207 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.165188 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.166623 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.167215 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.167367 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:16.171992 (+ 4625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:55911: BlockingRecv error: recv got EOF from 127.15.91.2:55911 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Jun 19 18:54:16 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:16 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.1@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:16.665818 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:16.653553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.1:41541)
0619 18:54:16.654169 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:16.654187 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:16.654309 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:16.654638 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:16.654641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:16.655084 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:16.655285 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:16.655294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.656379 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.656382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:16.657356 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:16.657371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.658200 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.658206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:16.658338 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:16.662279 (+ 3941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:16.662310 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:16.665531 (+ 3221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":347,"threads_started":1}
W20260619 18:54:16.666056 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:16.653705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:41541 (local address 127.15.91.62:44637)
0619 18:54:16.653920 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:16.653924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:16.654272 (+ 348us) server_negotiation.cc:407] Connection header received
0619 18:54:16.654418 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:16.654422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:16.654488 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:16.654584 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:16.655462 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.656225 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.657549 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.658007 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.658126 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:16.665784 (+ 7658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.1:41541: BlockingRecv error: recv got EOF from 127.15.91.1:41541 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 18:54:16 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:16 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.3@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:16.710398 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:16.701745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.3:49109)
0619 18:54:16.702241 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:16.702258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:16.702373 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:16.702704 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:16.702707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:16.703122 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:16.703306 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:16.703315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.704369 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.704373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:16.705323 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:16.705338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.706101 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.706105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:16.706233 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:16.706910 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:16.706929 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:16.710047 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":263,"threads_started":1}
W20260619 18:54:16.710565 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:16.701850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:49109 (local address 127.15.91.62:44637)
0619 18:54:16.702143 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:16.702150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:16.702345 (+ 195us) server_negotiation.cc:407] Connection header received
0619 18:54:16.702490 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:16.702494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:16.702575 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:16.702660 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:16.703464 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.704213 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.705490 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:16.705952 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:16.706056 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:16.710295 (+ 4239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.3:49109: BlockingRecv error: recv got EOF from 127.15.91.3:49109 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 18:54:17 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:54:17 dist-test-slave-dm3f krb5kdc[23981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.91.2@KRBTEST.COM for oryx/127.15.91.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:54:17.182672 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:17.173308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44637 (local address 127.15.91.2:56781)
0619 18:54:17.173897 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:17.173915 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:17.174092 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:17.174446 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:17.174451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:17.175056 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:17.175347 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:17.175363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:17.176497 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:17.176503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:17.177475 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:17.177491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:17.178342 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:17.178349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:17.178546 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:17.179274 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:17.179292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:17.182273 (+ 2981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.91.62:44637: Server oryx/127.15.91.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":340,"threads_started":1}
W20260619 18:54:17.182840 24719 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:17.173517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:56781 (local address 127.15.91.62:44637)
0619 18:54:17.173753 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:17.173760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:17.174014 (+ 254us) server_negotiation.cc:407] Connection header received
0619 18:54:17.174220 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:17.174226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:17.174301 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:17.174414 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:17.175502 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:17.176326 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:17.177662 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:17.178152 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:17.178312 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:17.182533 (+ 4221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.91.2:56781: BlockingRecv error: recv got EOF from 127.15.91.2:56781 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 18:54:17.510679 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 24570
I20260619 18:54:17.524262 25030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:17.511937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.15.91.1:38191)
0619 18:54:17.512545 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:17.512553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:17.512580 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:54:17.512643 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:17.512650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:17.512866 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:17.513051 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:17.514563 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:17.515752 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:17.517267 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:17.517881 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:17.518009 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:17.520714 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:17.520723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:17.520740 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:17.520784 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:17.523681 (+ 2897us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.91.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 18:54:17.524014 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42964: Request ticket server kudu/127.15.91.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":316,"threads_started":1}
W20260619 18:54:17.524410 25030 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42964: Request ticket server kudu/127.15.91.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 18:54:17.524663 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 24570: Not authorized: Client connection negotiation failed: client connection to 127.15.91.1:38191: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.91.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 18:54:17.524928 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 24570
I20260619 18:54:17.542899 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 24721
I20260619 18:54:17.558133 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:17.544121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.15.91.2:38217)
0619 18:54:17.544754 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:17.544769 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:17.544803 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:54:17.544865 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:17.544871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:17.545151 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:17.545330 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:17.546654 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:17.548228 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:17.550112 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:17.551134 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:17.551304 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:17.554445 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:17.554458 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:17.554476 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:17.554527 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:17.557545 (+ 3018us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.91.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 18:54:17.557875 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36692: Request ticket server kudu/127.15.91.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":301,"threads_started":1}
W20260619 18:54:17.558276 25031 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36692: Request ticket server kudu/127.15.91.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 18:54:17.558507 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 24721: Not authorized: Client connection negotiation failed: client connection to 127.15.91.2:38217: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.91.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 18:54:17.558703 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 24721
I20260619 18:54:17.576351 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 24852
I20260619 18:54:17.591666 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:17.577496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.15.91.3:45513)
0619 18:54:17.578057 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:17.578065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:17.578090 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:54:17.578172 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:17.578179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:17.578408 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:17.578583 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:17.580350 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:17.581501 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:17.583063 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:17.583688 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:17.583821 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:17.587918 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:17.587930 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:17.587953 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:17.588007 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:17.591057 (+ 3050us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.91.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 18:54:17.591406 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57824: Request ticket server kudu/127.15.91.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":279,"threads_started":1}
W20260619 18:54:17.591810 25032 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57824: Request ticket server kudu/127.15.91.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 18:54:17.592100 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 24852: Not authorized: Client connection negotiation failed: client connection to 127.15.91.3:45513: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.91.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 18:54:17.592293 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 24852
I20260619 18:54:17.609994 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 24501
I20260619 18:54:17.611451 24563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:17.731029 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 24501
2026-06-19T18:54:17Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21733 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T18:54:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:54:17Z Disabled control of system clock
I20260619 18:54:17.813020 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:41981
--webserver_interface=127.15.91.62
--webserver_port=0
--builtin_ntp_servers=127.15.91.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:41981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 18:54:18.180541 25037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:18.180958 25037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:18.181130 25037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:18.188772 25037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:54:18.188975 25037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:18.189054 25037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:18.189149 25037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:54:18.189218 25037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:54:18.199803 25037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:41981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:41981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:18.201278 25037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:18.203378 25037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:18.217985 25042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:18.218037 25043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:18.217939 25045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:18.223874 25037 server_base.cc:1034] running on GCE node
I20260619 18:54:18.224891 25037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:18.226604 25037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:18.227913 25037 hybrid_clock.cc:648] HybridClock initialized: now 1781895258227860 us; error 48 us; skew 500 ppm
I20260619 18:54:18.228354 25037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:54:18.231199 25037 webserver.cc:460] Webserver started at http://127.15.91.62:36389/ using document root <none> and password file <none>
I20260619 18:54:18.232048 25037 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:18.232209 25037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:18.232592 25037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:18.235965 25037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "c341841793a3467b843f47bd7e24aba5"
format_stamp: "Formatted at 2026-06-19 18:54:18 on dist-test-slave-dm3f"
server_key: "d1e1cc69618bcbc0464db9c99f66f60b"
server_key_iv: "1af8ab1af3271ac814f22e726f89f760"
server_key_version: "encryptionkey@0"
I20260619 18:54:18.236944 25037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "c341841793a3467b843f47bd7e24aba5"
format_stamp: "Formatted at 2026-06-19 18:54:18 on dist-test-slave-dm3f"
server_key: "d1e1cc69618bcbc0464db9c99f66f60b"
server_key_iv: "1af8ab1af3271ac814f22e726f89f760"
server_key_version: "encryptionkey@0"
I20260619 18:54:18.243372 25037 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:54:18.247684 25051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:18.249383 25037 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:54:18.249650 25037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "c341841793a3467b843f47bd7e24aba5"
format_stamp: "Formatted at 2026-06-19 18:54:18 on dist-test-slave-dm3f"
server_key: "d1e1cc69618bcbc0464db9c99f66f60b"
server_key_iv: "1af8ab1af3271ac814f22e726f89f760"
server_key_version: "encryptionkey@0"
I20260619 18:54:18.249929 25037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:18.291178 25037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:18.292193 25037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:18.292562 25037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:18.314042 25037 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:41981
I20260619 18:54:18.314061 25102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:41981 every 8 connection(s)
I20260619 18:54:18.315762 25037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:54:18.319640 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 25037
I20260619 18:54:18.319864 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:54:18.320736 15724 external_mini_cluster.cc:1428] Setting key fbcbe6434ba1e1ea6c6793e3b54cdc21
I20260619 18:54:18.321092 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:54:18.338570 25103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5: Bootstrap starting.
I20260619 18:54:18.339226 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:18.323349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.15.91.62:41981)
0619 18:54:18.325151 (+ 1802us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:18.325173 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:18.325270 (+ 97us) server_negotiation.cc:407] Connection header received
0619 18:54:18.327658 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:18.327700 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:18.328364 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:18.328833 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:18.329711 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:18.331337 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:18.333420 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:18.334251 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:18.334445 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:18.336246 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:18.336256 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:18.336275 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:18.336923 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:18.337046 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:18.337074 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:18.337187 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:18.337525 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 18:54:18.337972 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":471,"threads_started":1}
I20260619 18:54:18.342286 25103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5: Neither blocks nor log segments found. Creating new log.
I20260619 18:54:18.343590 25103 log.cc:824] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:54:18.347201 25103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5: No bootstrap required, opened a new log
I20260619 18:54:18.355448 25103 raft_consensus.cc:348] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c341841793a3467b843f47bd7e24aba5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41981 } }
I20260619 18:54:18.355944 25103 raft_consensus.cc:374] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:54:18.356058 25103 raft_consensus.cc:729] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c341841793a3467b843f47bd7e24aba5, State: Initialized, Role: FOLLOWER
I20260619 18:54:18.356765 25103 consensus_queue.cc:260] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c341841793a3467b843f47bd7e24aba5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41981 } }
I20260619 18:54:18.357059 25103 raft_consensus.cc:388] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:54:18.357218 25103 raft_consensus.cc:482] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:54:18.357426 25103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:54:18.359997 25103 raft_consensus.cc:504] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c341841793a3467b843f47bd7e24aba5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41981 } }
I20260619 18:54:18.360783 25103 leader_election.cc:302] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c341841793a3467b843f47bd7e24aba5; no voters:
I20260619 18:54:18.361528 25103 leader_election.cc:288] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:54:18.361756 25108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:54:18.362807 25108 raft_consensus.cc:686] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [term 1 LEADER]: Becoming Leader. State: Replica: c341841793a3467b843f47bd7e24aba5, State: Running, Role: LEADER
I20260619 18:54:18.363601 25108 consensus_queue.cc:237] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c341841793a3467b843f47bd7e24aba5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41981 } }
I20260619 18:54:18.364243 25103 sys_catalog.cc:564] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:54:18.367617 25109 sys_catalog.cc:455] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c341841793a3467b843f47bd7e24aba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c341841793a3467b843f47bd7e24aba5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41981 } } }
I20260619 18:54:18.367619 25110 sys_catalog.cc:455] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c341841793a3467b843f47bd7e24aba5. Latest consensus state: current_term: 1 leader_uuid: "c341841793a3467b843f47bd7e24aba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c341841793a3467b843f47bd7e24aba5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41981 } } }
I20260619 18:54:18.368160 25109 sys_catalog.cc:458] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:54:18.368227 25110 sys_catalog.cc:458] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:54:18.369359 25114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:54:18.377091 25114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:54:18.388651 25114 catalog_manager.cc:1347] Generated new cluster ID: aab50400246346c3b7d5cfd0f830e23f
I20260619 18:54:18.388862 25114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:54:18.404513 25114 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:54:18.406564 25114 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:54:18.425602 25114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c341841793a3467b843f47bd7e24aba5: Generated new TSK 0
I20260619 18:54:18.426929 25114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:54:18.439651 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:41981
--builtin_ntp_servers=127.15.91.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 18:54:18.805158 25127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:18.805497 25127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:18.805589 25127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:18.813578 25127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:18.813714 25127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:18.813823 25127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:54:18.823849 25127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:41981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:18.825176 25127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:18.827210 25127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:18.842563 25133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:18.842896 25132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:18.842890 25135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:18.843809 25127 server_base.cc:1034] running on GCE node
I20260619 18:54:18.844512 25127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:18.845705 25127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:18.846952 25127 hybrid_clock.cc:648] HybridClock initialized: now 1781895258846875 us; error 78 us; skew 500 ppm
I20260619 18:54:18.847289 25127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:54:18.850061 25127 webserver.cc:460] Webserver started at http://127.15.91.1:45689/ using document root <none> and password file <none>
I20260619 18:54:18.850971 25127 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:18.851089 25127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:18.851387 25127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:18.855043 25127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "0a1c310a0e5b4589be5f4a7d65aa64b5"
format_stamp: "Formatted at 2026-06-19 18:54:18 on dist-test-slave-dm3f"
server_key: "70d7113e6a63178115ca49ad365b6556"
server_key_iv: "75e2758f018ac60e719a35ed93d26cbb"
server_key_version: "encryptionkey@0"
I20260619 18:54:18.856053 25127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "0a1c310a0e5b4589be5f4a7d65aa64b5"
format_stamp: "Formatted at 2026-06-19 18:54:18 on dist-test-slave-dm3f"
server_key: "70d7113e6a63178115ca49ad365b6556"
server_key_iv: "75e2758f018ac60e719a35ed93d26cbb"
server_key_version: "encryptionkey@0"
I20260619 18:54:18.862447 25127 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:54:18.867278 25141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:18.869182 25127 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:54:18.869429 25127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "0a1c310a0e5b4589be5f4a7d65aa64b5"
format_stamp: "Formatted at 2026-06-19 18:54:18 on dist-test-slave-dm3f"
server_key: "70d7113e6a63178115ca49ad365b6556"
server_key_iv: "75e2758f018ac60e719a35ed93d26cbb"
server_key_version: "encryptionkey@0"
I20260619 18:54:18.869724 25127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:18.897146 25127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:18.898156 25127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:18.898495 25127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:18.899709 25127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:18.901364 25127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:18.901468 25127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:18.901646 25127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:18.901746 25127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:18.939019 25127 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:39613
I20260619 18:54:18.939119 25253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:39613 every 8 connection(s)
I20260619 18:54:18.940491 25127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:54:18.945080 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 25127
I20260619 18:54:18.945284 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:54:18.945986 15724 external_mini_cluster.cc:1428] Setting key 5afd3b1440493dab3fe063871c714f7c
I20260619 18:54:18.949214 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:41981
--builtin_ntp_servers=127.15.91.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 18:54:18.955564 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:18.943443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:57663 (local address 127.15.91.62:41981)
0619 18:54:18.943901 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:18.943906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:18.945554 (+ 1648us) server_negotiation.cc:407] Connection header received
0619 18:54:18.948030 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:18.948036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:18.948132 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:18.948221 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:18.949613 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:18.950346 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:18.952242 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:18.952667 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:18.952834 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:18.954499 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:18.954505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:18.954508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:18.954529 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:18.954599 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:18.954620 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:18.954761 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:18.955031 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 18:54:18.955146 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260619 18:54:18.956753 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:18.944130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:41981 (local address 127.15.91.1:57663)
0619 18:54:18.945285 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:18.945416 (+ 131us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:18.947625 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:18.948533 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:18.948542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:18.948660 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:18.949353 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:18.949390 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:18.950578 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:18.950583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:18.952025 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:18.952042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:18.952955 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:18.952965 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:18.953221 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:18.954060 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:54:18.954062 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:18.954109 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:54:18.954256 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:54:18.954762 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:54:18.954778 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:54:18.954789 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:54:18.955002 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 18:54:18.955507 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":376,"threads_started":1}
I20260619 18:54:18.958745 25254 heartbeater.cc:347] Connected to a master server at 127.15.91.62:41981
I20260619 18:54:18.959132 25254 heartbeater.cc:464] Registering TS with master...
I20260619 18:54:18.960197 25254 heartbeater.cc:511] Master 127.15.91.62:41981 requested a full tablet report, sending...
I20260619 18:54:18.963063 25068 ts_manager.cc:194] Registered new tserver with Master: 0a1c310a0e5b4589be5f4a7d65aa64b5 (127.15.91.1:39613)
I20260619 18:54:18.965555 25068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.1:57663
W20260619 18:54:19.312883 25258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:19.313225 25258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:19.313388 25258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:19.322031 25258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:19.322239 25258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:19.322432 25258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:54:19.333034 25258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:41981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:19.334496 25258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:19.336665 25258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:19.352151 25263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:19.352059 25266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:19.352092 25264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:19.353300 25258 server_base.cc:1034] running on GCE node
I20260619 18:54:19.354163 25258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:19.355535 25258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:19.356824 25258 hybrid_clock.cc:648] HybridClock initialized: now 1781895259356745 us; error 80 us; skew 500 ppm
I20260619 18:54:19.357236 25258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:54:19.360131 25258 webserver.cc:460] Webserver started at http://127.15.91.2:43617/ using document root <none> and password file <none>
I20260619 18:54:19.361004 25258 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:19.361176 25258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:19.361563 25258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:19.365052 25258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "cdc216452fb2477f822c9fba0eeb8290"
format_stamp: "Formatted at 2026-06-19 18:54:19 on dist-test-slave-dm3f"
server_key: "0bec6e4b237dfde3cd56a8381430bfaa"
server_key_iv: "7c4466638e3f0658501ce59a767cae8f"
server_key_version: "encryptionkey@0"
I20260619 18:54:19.366237 25258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "cdc216452fb2477f822c9fba0eeb8290"
format_stamp: "Formatted at 2026-06-19 18:54:19 on dist-test-slave-dm3f"
server_key: "0bec6e4b237dfde3cd56a8381430bfaa"
server_key_iv: "7c4466638e3f0658501ce59a767cae8f"
server_key_version: "encryptionkey@0"
I20260619 18:54:19.372514 25258 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260619 18:54:19.377331 25272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:19.379166 25258 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:54:19.379401 25258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "cdc216452fb2477f822c9fba0eeb8290"
format_stamp: "Formatted at 2026-06-19 18:54:19 on dist-test-slave-dm3f"
server_key: "0bec6e4b237dfde3cd56a8381430bfaa"
server_key_iv: "7c4466638e3f0658501ce59a767cae8f"
server_key_version: "encryptionkey@0"
I20260619 18:54:19.379653 25258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:19.399901 25258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:19.400900 25258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:19.401226 25258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:19.402352 25258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:19.404079 25258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:19.404181 25258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:19.404361 25258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:19.404446 25258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:19.442004 25258 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:39359
I20260619 18:54:19.442067 25384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:39359 every 8 connection(s)
I20260619 18:54:19.443645 25258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:54:19.447279 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 25258
I20260619 18:54:19.447425 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:54:19.447942 15724 external_mini_cluster.cc:1428] Setting key 21c644610957d7c9e77c82123e1a9580
I20260619 18:54:19.451224 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:41981
--builtin_ntp_servers=127.15.91.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 18:54:19.458707 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:19.446522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:58213 (local address 127.15.91.62:41981)
0619 18:54:19.446751 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:19.446760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:19.448763 (+ 2003us) server_negotiation.cc:407] Connection header received
0619 18:54:19.451108 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:19.451114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:19.451211 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:19.451339 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:19.452648 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:19.453393 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:19.455544 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:19.455991 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:19.456146 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:19.457796 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:19.457802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:19.457805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:19.457827 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:19.457861 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:19.457874 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:19.457984 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:19.458248 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 18:54:19.458363 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:54:19.459950 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:19.447174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:41981 (local address 127.15.91.2:58213)
0619 18:54:19.448569 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:19.448633 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:19.450661 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:19.451638 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:19.451649 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:19.451757 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:19.452428 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:19.452448 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:19.453590 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:19.453597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:19.455335 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:19.455356 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:19.456281 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:19.456312 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:19.456558 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:19.457379 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:54:19.457382 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:19.457429 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:54:19.457529 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:54:19.457993 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:54:19.458006 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:54:19.458018 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 18:54:19.458237 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 18:54:19.458725 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":525,"threads_started":1}
I20260619 18:54:19.461741 25385 heartbeater.cc:347] Connected to a master server at 127.15.91.62:41981
I20260619 18:54:19.462109 25385 heartbeater.cc:464] Registering TS with master...
I20260619 18:54:19.463105 25385 heartbeater.cc:511] Master 127.15.91.62:41981 requested a full tablet report, sending...
I20260619 18:54:19.464860 25068 ts_manager.cc:194] Registered new tserver with Master: cdc216452fb2477f822c9fba0eeb8290 (127.15.91.2:39359)
I20260619 18:54:19.465849 25068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.2:58213
W20260619 18:54:19.815055 25388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:19.815522 25388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:19.815689 25388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:19.823906 25388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:19.824080 25388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:19.824261 25388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:54:19.834089 25388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:41981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:19.835599 25388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:19.837584 25388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:19.852614 25393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:19.852699 25394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:19.852666 25396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:19.853739 25388 server_base.cc:1034] running on GCE node
I20260619 18:54:19.854590 25388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:19.856184 25388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:19.857492 25388 hybrid_clock.cc:648] HybridClock initialized: now 1781895259857366 us; error 166 us; skew 500 ppm
I20260619 18:54:19.857903 25388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:54:19.860674 25388 webserver.cc:460] Webserver started at http://127.15.91.3:37951/ using document root <none> and password file <none>
I20260619 18:54:19.861438 25388 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:19.861553 25388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:19.861846 25388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:19.865339 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "f40aeda904a544b0a833b424f2ee8458"
format_stamp: "Formatted at 2026-06-19 18:54:19 on dist-test-slave-dm3f"
server_key: "1d0947c5dde7c50f4d82c808be7cfa53"
server_key_iv: "1715ef7aa3b4a27a4bde68db9a7709d1"
server_key_version: "encryptionkey@0"
I20260619 18:54:19.866300 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "f40aeda904a544b0a833b424f2ee8458"
format_stamp: "Formatted at 2026-06-19 18:54:19 on dist-test-slave-dm3f"
server_key: "1d0947c5dde7c50f4d82c808be7cfa53"
server_key_iv: "1715ef7aa3b4a27a4bde68db9a7709d1"
server_key_version: "encryptionkey@0"
I20260619 18:54:19.872884 25388 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:54:19.878011 25402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:19.879889 25388 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:54:19.880134 25388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "f40aeda904a544b0a833b424f2ee8458"
format_stamp: "Formatted at 2026-06-19 18:54:19 on dist-test-slave-dm3f"
server_key: "1d0947c5dde7c50f4d82c808be7cfa53"
server_key_iv: "1715ef7aa3b4a27a4bde68db9a7709d1"
server_key_version: "encryptionkey@0"
I20260619 18:54:19.880393 25388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:19.898319 25388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:19.899390 25388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:19.899724 25388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:19.900862 25388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:19.902595 25388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:19.902706 25388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:19.902902 25388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:19.903000 25388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:19.940563 25388 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:41171
I20260619 18:54:19.940595 25514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:41171 every 8 connection(s)
I20260619 18:54:19.942390 25388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:54:19.948309 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 25388
I20260619 18:54:19.948508 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:54:19.949069 15724 external_mini_cluster.cc:1428] Setting key 37236deff7cdef2567a8e2229456d079
I20260619 18:54:19.958251 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:19.945748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:59087 (local address 127.15.91.62:41981)
0619 18:54:19.945943 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:19.945948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:19.947744 (+ 1796us) server_negotiation.cc:407] Connection header received
0619 18:54:19.950508 (+ 2764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:19.950514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:19.950612 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:19.950778 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:19.952087 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:19.952929 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:19.954939 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:19.955381 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:19.955562 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:19.957267 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:19.957272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:19.957276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:19.957299 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:19.957333 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:19.957348 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:19.957460 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:19.957720 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 18:54:19.957839 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:54:19.959460 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:19.946397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:41981 (local address 127.15.91.3:59087)
0619 18:54:19.947543 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:19.947586 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:19.950124 (+ 2538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:19.951072 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:19.951082 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:19.951193 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:19.951879 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:19.951900 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:19.953098 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:19.953102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:19.954686 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:19.954708 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:19.955667 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:19.955674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:19.955951 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:19.956823 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:54:19.956826 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:19.956873 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:54:19.956974 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:54:19.957454 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:54:19.957466 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:54:19.957478 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 18:54:19.957694 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 18:54:19.958188 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":390,"threads_started":1}
I20260619 18:54:19.961176 25515 heartbeater.cc:347] Connected to a master server at 127.15.91.62:41981
I20260619 18:54:19.961479 25515 heartbeater.cc:464] Registering TS with master...
I20260619 18:54:19.962301 25515 heartbeater.cc:511] Master 127.15.91.62:41981 requested a full tablet report, sending...
I20260619 18:54:19.964054 25067 ts_manager.cc:194] Registered new tserver with Master: f40aeda904a544b0a833b424f2ee8458 (127.15.91.3:41171)
I20260619 18:54:19.965030 25067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.3:59087
I20260619 18:54:19.965462 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:54:19.970563 25254 heartbeater.cc:503] Master 127.15.91.62:41981 was elected leader, sending a full tablet report...
W20260619 18:54:19.972720 25256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:19.970620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.15.91.62:41981)
0619 18:54:19.970913 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:19.970921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:19.971534 (+ 613us) server_negotiation.cc:407] Connection header received
0619 18:54:19.971760 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:19.971767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:19.971838 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:19.971976 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:19.972472 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59310: BlockingRecv error: recv got EOF from 127.0.0.1:59310 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
I20260619 18:54:19.973876 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 25127
I20260619 18:54:19.982821 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:19.975048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.15.91.1:39613)
0619 18:54:19.975654 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:19.975661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:19.975688 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:54:19.975747 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:19.975754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:19.975996 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:19.976202 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:19.976603 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:19.977753 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:19.979189 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:19.979871 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:19.979990 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:19.981179 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:19.981187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:19.981202 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:19.981834 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:19.981892 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:19.981915 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:19.982023 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:19.982190 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 18:54:19.982370 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":276,"threads_started":1}
I20260619 18:54:19.984261 25249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:20.072736 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 25127
I20260619 18:54:20.093418 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 25258
I20260619 18:54:20.105314 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:20.094766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.15.91.2:39359)
0619 18:54:20.095330 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:20.095345 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:20.095375 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:54:20.095446 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:20.095454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:20.095693 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:20.095911 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:20.096427 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:20.097758 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:20.100108 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:20.100997 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:20.101183 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:20.102774 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:20.102788 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:20.102836 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:20.103953 (+ 1117us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:20.104069 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:20.104106 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:20.104301 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:20.104508 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 18:54:20.104769 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":293,"threads_started":1}
I20260619 18:54:20.107198 25380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:20.192360 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 25258
I20260619 18:54:20.209307 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 25388
I20260619 18:54:20.218453 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:20.210283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.15.91.3:41171)
0619 18:54:20.210987 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:20.210995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:20.211030 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:54:20.211100 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:20.211106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:20.211326 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:20.211472 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:20.211947 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:20.213064 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:20.214477 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:20.215408 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:20.215551 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:20.216834 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:20.216844 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:20.216859 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:20.217495 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:20.217553 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:20.217578 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:20.217739 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:20.217946 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 18:54:20.218121 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":360,"threads_started":1}
I20260619 18:54:20.219966 25510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:20.305455 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 25388
I20260619 18:54:20.322202 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 25037
I20260619 18:54:20.323566 25098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:20.428573 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 25037
2026-06-19T18:54:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2691 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T18:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:54:20Z Disabled control of system clock
I20260619 18:54:20.505489 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:42671
--webserver_interface=127.15.91.62
--webserver_port=0
--builtin_ntp_servers=127.15.91.20:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:42671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:54:20.873891 25534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:20.874231 25534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:20.874387 25534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:20.882794 25534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:54:20.882931 25534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:20.883038 25534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:20.883141 25534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:54:20.883239 25534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:54:20.893896 25534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:42671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:42671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:20.895804 25534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:20.898535 25534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:20.911863 25539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:20.911918 25542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:20.911900 25540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:20.912981 25534 server_base.cc:1034] running on GCE node
I20260619 18:54:20.913905 25534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:20.915755 25534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:20.917057 25534 hybrid_clock.cc:648] HybridClock initialized: now 1781895260916994 us; error 65 us; skew 500 ppm
I20260619 18:54:20.917516 25534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:54:20.920535 25534 webserver.cc:460] Webserver started at http://127.15.91.62:46611/ using document root <none> and password file <none>
I20260619 18:54:20.921425 25534 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:20.921593 25534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:20.921973 25534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:20.925436 25534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "8520dee0036c41258bb060d6699b1b52"
format_stamp: "Formatted at 2026-06-19 18:54:20 on dist-test-slave-dm3f"
server_key: "2ea4f39b78c308aea515b64689db4c39"
server_key_iv: "6d9b04ed109051c526a3e948b7d1d6b8"
server_key_version: "encryptionkey@0"
I20260619 18:54:20.926431 25534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "8520dee0036c41258bb060d6699b1b52"
format_stamp: "Formatted at 2026-06-19 18:54:20 on dist-test-slave-dm3f"
server_key: "2ea4f39b78c308aea515b64689db4c39"
server_key_iv: "6d9b04ed109051c526a3e948b7d1d6b8"
server_key_version: "encryptionkey@0"
I20260619 18:54:20.933609 25534 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260619 18:54:20.939921 25548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:20.941870 25534 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:54:20.942116 25534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "8520dee0036c41258bb060d6699b1b52"
format_stamp: "Formatted at 2026-06-19 18:54:20 on dist-test-slave-dm3f"
server_key: "2ea4f39b78c308aea515b64689db4c39"
server_key_iv: "6d9b04ed109051c526a3e948b7d1d6b8"
server_key_version: "encryptionkey@0"
I20260619 18:54:20.942391 25534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:20.949155 25534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:20.950137 25534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:20.950553 25534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:20.971236 25534 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:42671
I20260619 18:54:20.971251 25599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:42671 every 8 connection(s)
I20260619 18:54:20.972790 25534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:54:20.977265 25600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:54:20.981447 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 25534
I20260619 18:54:20.981685 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:54:20.982371 15724 external_mini_cluster.cc:1428] Setting key 048ed9b152e922848f3f9c6ca3f16613
I20260619 18:54:20.990370 25600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52: Bootstrap starting.
I20260619 18:54:20.992645 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:20.984879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.15.91.62:42671)
0619 18:54:20.986093 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:20.986106 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:20.986164 (+ 58us) server_negotiation.cc:407] Connection header received
0619 18:54:20.988261 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:20.988296 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:20.988857 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:20.989278 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:20.990664 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:20.990671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:20.990691 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:20.990767 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:20.990861 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:20.990882 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:20.991090 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:20.991208 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 18:54:20.991562 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":425,"threads_started":1}
I20260619 18:54:20.994238 25600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52: Neither blocks nor log segments found. Creating new log.
I20260619 18:54:20.995627 25600 log.cc:824] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52: Log is configured to *not* fsync() on all Append() calls
I20260619 18:54:20.999270 25600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52: No bootstrap required, opened a new log
I20260619 18:54:21.007643 25600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8520dee0036c41258bb060d6699b1b52" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42671 } }
I20260619 18:54:21.008116 25600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:54:21.008237 25600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8520dee0036c41258bb060d6699b1b52, State: Initialized, Role: FOLLOWER
I20260619 18:54:21.008908 25600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8520dee0036c41258bb060d6699b1b52" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42671 } }
I20260619 18:54:21.009184 25600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:54:21.009359 25600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:54:21.009555 25600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:54:21.012133 25600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8520dee0036c41258bb060d6699b1b52" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42671 } }
I20260619 18:54:21.012949 25600 leader_election.cc:302] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8520dee0036c41258bb060d6699b1b52; no voters:
I20260619 18:54:21.013679 25600 leader_election.cc:288] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:54:21.013849 25605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:54:21.015266 25605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [term 1 LEADER]: Becoming Leader. State: Replica: 8520dee0036c41258bb060d6699b1b52, State: Running, Role: LEADER
I20260619 18:54:21.015964 25605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8520dee0036c41258bb060d6699b1b52" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42671 } }
I20260619 18:54:21.016489 25600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:54:21.019438 25607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8520dee0036c41258bb060d6699b1b52. Latest consensus state: current_term: 1 leader_uuid: "8520dee0036c41258bb060d6699b1b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8520dee0036c41258bb060d6699b1b52" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42671 } } }
I20260619 18:54:21.019990 25607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [sys.catalog]: This master's current role is: LEADER
I20260619 18:54:21.019488 25606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8520dee0036c41258bb060d6699b1b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8520dee0036c41258bb060d6699b1b52" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42671 } } }
I20260619 18:54:21.020149 25606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52 [sys.catalog]: This master's current role is: LEADER
I20260619 18:54:21.024041 25614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:54:21.030378 25614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:54:21.042121 25614 catalog_manager.cc:1347] Generated new cluster ID: 5052d2b3a2a149349a2c6aa15a5e8eeb
I20260619 18:54:21.042265 25614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:54:21.058636 25614 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:54:21.059221 25614 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:54:21.074985 25614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8520dee0036c41258bb060d6699b1b52: Generated new TSK 0
I20260619 18:54:21.075973 25614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:54:21.085229 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42671
--builtin_ntp_servers=127.15.91.20:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:54:21.467396 25624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:21.467731 25624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:21.467919 25624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:21.476308 25624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:21.476447 25624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:21.476655 25624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:54:21.486550 25624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:21.488021 25624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:21.490018 25624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:21.505082 25630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:21.505331 25629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:21.505329 25632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:21.506381 25624 server_base.cc:1034] running on GCE node
I20260619 18:54:21.507155 25624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:21.508244 25624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:21.509533 25624 hybrid_clock.cc:648] HybridClock initialized: now 1781895261509478 us; error 52 us; skew 500 ppm
I20260619 18:54:21.509922 25624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:54:21.512619 25624 webserver.cc:460] Webserver started at http://127.15.91.1:36261/ using document root <none> and password file <none>
I20260619 18:54:21.513477 25624 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:21.513639 25624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:21.514008 25624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:21.517536 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "44c41684912041d4a4735a64f84adefc"
format_stamp: "Formatted at 2026-06-19 18:54:21 on dist-test-slave-dm3f"
server_key: "11629ee62a1f3af8c9d180cbf01b8f28"
server_key_iv: "0e119dfe6237102bf78b104da55535cd"
server_key_version: "encryptionkey@0"
I20260619 18:54:21.518640 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "44c41684912041d4a4735a64f84adefc"
format_stamp: "Formatted at 2026-06-19 18:54:21 on dist-test-slave-dm3f"
server_key: "11629ee62a1f3af8c9d180cbf01b8f28"
server_key_iv: "0e119dfe6237102bf78b104da55535cd"
server_key_version: "encryptionkey@0"
I20260619 18:54:21.525347 25624 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260619 18:54:21.529852 25638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:21.531834 25624 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:54:21.532087 25624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "44c41684912041d4a4735a64f84adefc"
format_stamp: "Formatted at 2026-06-19 18:54:21 on dist-test-slave-dm3f"
server_key: "11629ee62a1f3af8c9d180cbf01b8f28"
server_key_iv: "0e119dfe6237102bf78b104da55535cd"
server_key_version: "encryptionkey@0"
I20260619 18:54:21.532387 25624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:21.539152 25624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:21.540205 25624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:21.540552 25624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:21.541652 25624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:21.543524 25624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:21.543674 25624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:21.543864 25624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:21.543962 25624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:21.583532 25624 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:34991
I20260619 18:54:21.583559 25750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:34991 every 8 connection(s)
I20260619 18:54:21.585062 25624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:54:21.594110 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 25624
I20260619 18:54:21.594380 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:54:21.595148 15724 external_mini_cluster.cc:1428] Setting key 3b48b4cc003510d2e3fbaae1da31a502
I20260619 18:54:21.596864 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:21.588486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:52071 (local address 127.15.91.62:42671)
0619 18:54:21.589026 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:21.589031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:21.590548 (+ 1517us) server_negotiation.cc:407] Connection header received
0619 18:54:21.593045 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:21.593053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:21.593173 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:21.593348 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:21.595781 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:21.595788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:21.595792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:21.595822 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:21.595898 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:21.595914 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:21.596094 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:21.596406 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 18:54:21.596495 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":272,"threads_started":1}
I20260619 18:54:21.598098 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:21.589204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42671 (local address 127.15.91.1:52071)
0619 18:54:21.590330 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:21.590383 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:21.592226 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:21.593748 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:21.593762 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:21.593937 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:21.595016 (+ 1079us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:54:21.595019 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:21.595083 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:54:21.595258 (+ 175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:54:21.596058 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:54:21.596075 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:54:21.596085 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:54:21.596529 (+ 444us) client_negotiation.cc:240] Negotiation successful
0619 18:54:21.596914 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":357,"threads_started":1}
I20260619 18:54:21.599355 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42671
--builtin_ntp_servers=127.15.91.20:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 18:54:21.600528 25751 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42671
I20260619 18:54:21.601212 25751 heartbeater.cc:464] Registering TS with master...
I20260619 18:54:21.602155 25751 heartbeater.cc:511] Master 127.15.91.62:42671 requested a full tablet report, sending...
I20260619 18:54:21.605647 25565 ts_manager.cc:194] Registered new tserver with Master: 44c41684912041d4a4735a64f84adefc (127.15.91.1:34991)
W20260619 18:54:21.960598 25755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:21.960927 25755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:21.961091 25755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:21.968972 25755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:21.969113 25755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:21.969311 25755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:54:21.984521 25755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:21.986251 25755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:21.988806 25755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:22.004745 25760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:22.004702 25763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:22.004797 25761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:22.008399 25755 server_base.cc:1034] running on GCE node
I20260619 18:54:22.009116 25755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:22.010215 25755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:22.011451 25755 hybrid_clock.cc:648] HybridClock initialized: now 1781895262011393 us; error 60 us; skew 500 ppm
I20260619 18:54:22.011797 25755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:54:22.018532 25755 webserver.cc:460] Webserver started at http://127.15.91.2:40099/ using document root <none> and password file <none>
I20260619 18:54:22.019467 25755 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:22.019587 25755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:22.019882 25755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:22.023586 25755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "6a1ddbd428974512864170862f830514"
format_stamp: "Formatted at 2026-06-19 18:54:22 on dist-test-slave-dm3f"
server_key: "0bc12847b5a704c56ceb31fb6891ab20"
server_key_iv: "51042e17530901c92d4a63f77d2efd92"
server_key_version: "encryptionkey@0"
I20260619 18:54:22.024513 25755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "6a1ddbd428974512864170862f830514"
format_stamp: "Formatted at 2026-06-19 18:54:22 on dist-test-slave-dm3f"
server_key: "0bc12847b5a704c56ceb31fb6891ab20"
server_key_iv: "51042e17530901c92d4a63f77d2efd92"
server_key_version: "encryptionkey@0"
I20260619 18:54:22.031044 25755 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:54:22.035676 25769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:22.037657 25755 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:54:22.037859 25755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "6a1ddbd428974512864170862f830514"
format_stamp: "Formatted at 2026-06-19 18:54:22 on dist-test-slave-dm3f"
server_key: "0bc12847b5a704c56ceb31fb6891ab20"
server_key_iv: "51042e17530901c92d4a63f77d2efd92"
server_key_version: "encryptionkey@0"
I20260619 18:54:22.038185 25755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:22.045054 25755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:22.046079 25755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:22.046475 25755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:22.047828 25755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:22.049602 25755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:22.049714 25755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:22.049902 25755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:22.049970 25755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:22.088392 25755 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:45611
I20260619 18:54:22.088523 25881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:45611 every 8 connection(s)
I20260619 18:54:22.089960 25755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:54:22.099179 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 25755
I20260619 18:54:22.099404 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:54:22.100000 15724 external_mini_cluster.cc:1428] Setting key 21eb026d9f8d2eef46c11bd142bb810a
I20260619 18:54:22.100026 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:22.093132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:47455 (local address 127.15.91.62:42671)
0619 18:54:22.093379 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:22.093386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:22.095171 (+ 1785us) server_negotiation.cc:407] Connection header received
0619 18:54:22.097092 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:22.097097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:22.097166 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:22.097281 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:22.099059 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:22.099066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:22.099071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:22.099100 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:22.099150 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:22.099169 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:22.099330 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:22.099633 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 18:54:22.099736 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 18:54:22.101332 25884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:22.093749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42671 (local address 127.15.91.2:47455)
0619 18:54:22.094967 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:22.095018 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:22.096700 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:22.097573 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:22.097582 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:22.097697 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:22.098514 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:54:22.098517 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:22.098564 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:54:22.098732 (+ 168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:54:22.099315 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:54:22.099331 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:54:22.099342 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:54:22.099737 (+ 395us) client_negotiation.cc:240] Negotiation successful
0619 18:54:22.100096 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":345,"threads_started":1}
I20260619 18:54:22.103075 25882 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42671
I20260619 18:54:22.103459 25882 heartbeater.cc:464] Registering TS with master...
I20260619 18:54:22.103716 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42671
--builtin_ntp_servers=127.15.91.20:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 18:54:22.104233 25882 heartbeater.cc:511] Master 127.15.91.62:42671 requested a full tablet report, sending...
I20260619 18:54:22.105965 25565 ts_manager.cc:194] Registered new tserver with Master: 6a1ddbd428974512864170862f830514 (127.15.91.2:45611)
W20260619 18:54:22.469292 25885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:22.469633 25885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:22.469727 25885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:22.477555 25885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:22.477695 25885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:22.477900 25885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:54:22.487771 25885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:22.489198 25885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:22.491295 25885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:22.506067 25891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:22.506152 25890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:22.506649 25893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:22.506740 25885 server_base.cc:1034] running on GCE node
I20260619 18:54:22.507457 25885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:22.508563 25885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:22.509835 25885 hybrid_clock.cc:648] HybridClock initialized: now 1781895262509781 us; error 69 us; skew 500 ppm
I20260619 18:54:22.510224 25885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:54:22.512930 25885 webserver.cc:460] Webserver started at http://127.15.91.3:42545/ using document root <none> and password file <none>
I20260619 18:54:22.513772 25885 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:22.513957 25885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:22.514326 25885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:22.517979 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "c1c0d7504ef648e6a63616c1090a55bb"
format_stamp: "Formatted at 2026-06-19 18:54:22 on dist-test-slave-dm3f"
server_key: "942eacffeb20861139ca6bf4e8f25ef5"
server_key_iv: "5da22b9d3d29bd09f81bb2a57d81f50a"
server_key_version: "encryptionkey@0"
I20260619 18:54:22.519094 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "c1c0d7504ef648e6a63616c1090a55bb"
format_stamp: "Formatted at 2026-06-19 18:54:22 on dist-test-slave-dm3f"
server_key: "942eacffeb20861139ca6bf4e8f25ef5"
server_key_iv: "5da22b9d3d29bd09f81bb2a57d81f50a"
server_key_version: "encryptionkey@0"
I20260619 18:54:22.525635 25885 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 18:54:22.530049 25899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:22.531801 25885 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260619 18:54:22.532050 25885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "c1c0d7504ef648e6a63616c1090a55bb"
format_stamp: "Formatted at 2026-06-19 18:54:22 on dist-test-slave-dm3f"
server_key: "942eacffeb20861139ca6bf4e8f25ef5"
server_key_iv: "5da22b9d3d29bd09f81bb2a57d81f50a"
server_key_version: "encryptionkey@0"
I20260619 18:54:22.532325 25885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:22.539417 25885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:22.540441 25885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:22.540776 25885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:22.541975 25885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:22.543843 25885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:22.543998 25885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:22.544147 25885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:22.544236 25885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:22.585031 25885 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:33697
I20260619 18:54:22.585053 26011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:33697 every 8 connection(s)
I20260619 18:54:22.586623 25885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:54:22.590698 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 25885
I20260619 18:54:22.590988 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:54:22.591486 15724 external_mini_cluster.cc:1428] Setting key be0486d5c10aac3b13e041dec2d874df
I20260619 18:54:22.597657 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:22.590490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:60663 (local address 127.15.91.62:42671)
0619 18:54:22.590766 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:22.590773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:22.592815 (+ 2042us) server_negotiation.cc:407] Connection header received
0619 18:54:22.594950 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:22.594956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:22.595036 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:22.595152 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:22.596832 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:22.596839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:22.596843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:22.596882 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:22.596921 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:22.596934 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:22.597075 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:22.597289 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 18:54:22.597368 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 18:54:22.598886 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:22.591263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42671 (local address 127.15.91.3:60663)
0619 18:54:22.592619 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:22.592679 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:22.594532 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:22.595422 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:22.595432 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:22.595549 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:22.596381 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:54:22.596383 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:22.596453 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:54:22.596565 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:54:22.597043 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:54:22.597054 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:54:22.597063 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:54:22.597404 (+ 341us) client_negotiation.cc:240] Negotiation successful
0619 18:54:22.597735 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":468,"threads_started":1}
I20260619 18:54:22.601001 26012 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42671
I20260619 18:54:22.601433 26012 heartbeater.cc:464] Registering TS with master...
I20260619 18:54:22.602185 26012 heartbeater.cc:511] Master 127.15.91.62:42671 requested a full tablet report, sending...
I20260619 18:54:22.604226 25565 ts_manager.cc:194] Registered new tserver with Master: c1c0d7504ef648e6a63616c1090a55bb (127.15.91.3:33697)
I20260619 18:54:22.608475 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:54:22.609896 25751 heartbeater.cc:503] Master 127.15.91.62:42671 was elected leader, sending a full tablet report...
W20260619 18:54:22.615792 25753 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:54:22.613975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.15.91.62:42671)
0619 18:54:22.614170 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:22.614175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:22.614855 (+ 680us) server_negotiation.cc:407] Connection header received
0619 18:54:22.615023 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:22.615027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:22.615089 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:22.615187 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:22.615510 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50298: BlockingRecv error: recv got EOF from 127.0.0.1:50298 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:54:22.616784 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 25624
I20260619 18:54:22.621117 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:22.618106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.15.91.1:34991)
0619 18:54:22.618818 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:22.618826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:22.618857 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:54:22.618925 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:22.618932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:22.619159 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:22.619312 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:22.620362 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:22.620369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:22.620381 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:22.620397 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:22.620446 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:22.620462 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:22.620613 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:22.620661 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 18:54:22.620752 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":336,"threads_started":1}
I20260619 18:54:22.622699 25746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:22.707269 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 25624
I20260619 18:54:22.725342 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 25755
I20260619 18:54:22.729350 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:22.726473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.15.91.2:45611)
0619 18:54:22.727165 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:22.727172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:22.727201 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:54:22.727268 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:22.727274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:22.727508 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:22.727649 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:22.728699 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:22.728706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:22.728718 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:22.728739 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:22.728783 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:22.728798 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:22.728932 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:22.729011 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 18:54:22.729095 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":393,"threads_started":1}
I20260619 18:54:22.730962 25877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:22.814497 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 25755
I20260619 18:54:22.832036 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 25885
I20260619 18:54:22.836570 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:22.833074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.15.91.3:33697)
0619 18:54:22.833774 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:22.833782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:22.833817 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:54:22.833882 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:22.833889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:22.834123 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:22.834353 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:22.835686 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:22.835693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:22.835707 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:54:22.835725 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:22.835773 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:54:22.835788 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:22.836006 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:22.836090 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 18:54:22.836229 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":371,"threads_started":1}
I20260619 18:54:22.838291 26007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:22.925189 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 25885
I20260619 18:54:22.942193 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 25534
I20260619 18:54:22.943555 25595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:23.048146 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 25534
2026-06-19T18:54:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2619 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:54:23 dist-test-slave-dm3f krb5kdc[26030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:54:23 dist-test-slave-dm3f krb5kdc[26030](info): set up 2 sockets
Jun 19 18:54:23 dist-test-slave-dm3f krb5kdc[26030](info): commencing operation
krb5kdc: starting...
W20260619 18:54:24.176815 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:54:24 dist-test-slave-dm3f krb5kdc[26030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:54:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:54:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:54:24.439025 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:44771
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:44771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:54:24.809955 26043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:24.810293 26043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:24.810405 26043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:24.818358 26043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:54:24.818502 26043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:24.818550 26043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:24.818600 26043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:54:24.818637 26043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:54:24.829511 26043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:44771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:44771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:24.830909 26043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:24.832294 26043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:24.846549 26048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:24.846673 26049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:24.846840 26051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:24.848086 26043 server_base.cc:1034] running on GCE node
I20260619 18:54:24.849155 26043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:24.850973 26043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:24.852325 26043 hybrid_clock.cc:648] HybridClock initialized: now 1781895264852246 us; error 74 us; skew 500 ppm
I20260619 18:54:24.852824 26043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:54:24 dist-test-slave-dm3f krb5kdc[26030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895264, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:54:24.856945 26043 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:54:24.858976 26043 webserver.cc:460] Webserver started at http://127.15.91.62:40469/ using document root <none> and password file <none>
I20260619 18:54:24.859856 26043 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:24.860044 26043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:24.860409 26043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:24.864020 26043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "cb446b2bb69644af8f5699c722d16cc8"
format_stamp: "Formatted at 2026-06-19 18:54:24 on dist-test-slave-dm3f"
server_key: "fecfdb27b984f92134b9e9befbeb3434"
server_key_iv: "d7a24e8734010820453bbef0eb586d08"
server_key_version: "encryptionkey@0"
I20260619 18:54:24.865031 26043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "cb446b2bb69644af8f5699c722d16cc8"
format_stamp: "Formatted at 2026-06-19 18:54:24 on dist-test-slave-dm3f"
server_key: "fecfdb27b984f92134b9e9befbeb3434"
server_key_iv: "d7a24e8734010820453bbef0eb586d08"
server_key_version: "encryptionkey@0"
I20260619 18:54:24.871645 26043 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:54:24.876670 26058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:24.878619 26043 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:54:24.878952 26043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "cb446b2bb69644af8f5699c722d16cc8"
format_stamp: "Formatted at 2026-06-19 18:54:24 on dist-test-slave-dm3f"
server_key: "fecfdb27b984f92134b9e9befbeb3434"
server_key_iv: "d7a24e8734010820453bbef0eb586d08"
server_key_version: "encryptionkey@0"
I20260619 18:54:24.879222 26043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:24.896978 26043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:24.902448 26043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:24.902876 26043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:24.927088 26043 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:44771
I20260619 18:54:24.927156 26109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:44771 every 8 connection(s)
I20260619 18:54:24.928817 26043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:54:24.934031 26110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:54:24.936712 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 26043
I20260619 18:54:24.936991 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:54:24.937808 15724 external_mini_cluster.cc:1428] Setting key d4e5f10d93aed30b1e93c394d1c11e1e
I20260619 18:54:24.949115 26110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8: Bootstrap starting.
I20260619 18:54:24.953624 26110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8: Neither blocks nor log segments found. Creating new log.
I20260619 18:54:24.955353 26110 log.cc:824] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:54:24 dist-test-slave-dm3f krb5kdc[26030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:54:24.959762 26110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8: No bootstrap required, opened a new log
I20260619 18:54:24.967612 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:24.940930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.15.91.62:44771)
0619 18:54:24.942182 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:24.942198 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:24.942256 (+ 58us) server_negotiation.cc:407] Connection header received
0619 18:54:24.944284 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:24.944317 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:24.944867 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:24.945313 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:24.947557 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:24.949409 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:24.951764 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:24.953048 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:24.953329 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:24.958824 (+ 5495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:24.958835 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:24.958853 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:24.958902 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:24.962442 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:24.963440 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:24.963456 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:24.963464 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:24.963682 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:24.964169 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:24.964173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:24.964175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:24.964881 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:24.965276 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:24.965782 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 18:54:24.966298 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":591,"threads_started":1}
I20260619 18:54:24.968838 26110 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb446b2bb69644af8f5699c722d16cc8" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44771 } }
I20260619 18:54:24.969390 26110 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:54:24.969511 26110 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb446b2bb69644af8f5699c722d16cc8, State: Initialized, Role: FOLLOWER
I20260619 18:54:24.970314 26110 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb446b2bb69644af8f5699c722d16cc8" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44771 } }
I20260619 18:54:24.970626 26110 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:54:24.970844 26110 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:54:24.971055 26110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:54:24.973552 26110 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb446b2bb69644af8f5699c722d16cc8" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44771 } }
I20260619 18:54:24.974172 26110 leader_election.cc:302] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb446b2bb69644af8f5699c722d16cc8; no voters:
I20260619 18:54:24.974824 26110 leader_election.cc:288] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:54:24.975006 26115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:54:24.976418 26115 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [term 1 LEADER]: Becoming Leader. State: Replica: cb446b2bb69644af8f5699c722d16cc8, State: Running, Role: LEADER
I20260619 18:54:24.977077 26115 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb446b2bb69644af8f5699c722d16cc8" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44771 } }
I20260619 18:54:24.977555 26110 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:54:24.981328 26117 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb446b2bb69644af8f5699c722d16cc8. Latest consensus state: current_term: 1 leader_uuid: "cb446b2bb69644af8f5699c722d16cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb446b2bb69644af8f5699c722d16cc8" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44771 } } }
I20260619 18:54:24.981894 26117 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:54:24.981343 26116 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb446b2bb69644af8f5699c722d16cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb446b2bb69644af8f5699c722d16cc8" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 44771 } } }
I20260619 18:54:24.982120 26116 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:54:24.983266 26122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:54:24.990401 26122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:54:25.001674 26122 catalog_manager.cc:1347] Generated new cluster ID: a60f503d54034c4b8d0fc3fada9f9f6b
I20260619 18:54:25.001835 26122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:54:25.012650 26122 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:54:25.014230 26122 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:54:25.022214 26122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8: Generated new TSK 0
I20260619 18:54:25.023279 26122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:54:25.088143 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44771
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:54:25.458563 26136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:25.459026 26136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:25.459218 26136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:25.467243 26136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:25.467382 26136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:25.467592 26136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:54:25.478494 26136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:25.480072 26136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:25.481608 26136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:25.497153 26144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:25.497180 26141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:25.497157 26142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:25.498461 26136 server_base.cc:1034] running on GCE node
I20260619 18:54:25.499313 26136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:25.500425 26136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:25.501703 26136 hybrid_clock.cc:648] HybridClock initialized: now 1781895265501631 us; error 73 us; skew 500 ppm
I20260619 18:54:25.502096 26136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:54:25 dist-test-slave-dm3f krb5kdc[26030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895265, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:54:25.505805 26136 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:54:25.507524 26136 webserver.cc:460] Webserver started at http://127.15.91.1:36405/ using document root <none> and password file <none>
I20260619 18:54:25.508301 26136 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:25.508433 26136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:25.508714 26136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:25.512049 26136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "ca3d5ce166f84170be37988787f69ed6"
format_stamp: "Formatted at 2026-06-19 18:54:25 on dist-test-slave-dm3f"
server_key: "475b8639d9a3c27f70ac6721a340d030"
server_key_iv: "d87af39ff76de5c44ba22d1b67a19cb0"
server_key_version: "encryptionkey@0"
I20260619 18:54:25.512910 26136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "ca3d5ce166f84170be37988787f69ed6"
format_stamp: "Formatted at 2026-06-19 18:54:25 on dist-test-slave-dm3f"
server_key: "475b8639d9a3c27f70ac6721a340d030"
server_key_iv: "d87af39ff76de5c44ba22d1b67a19cb0"
server_key_version: "encryptionkey@0"
I20260619 18:54:25.519237 26136 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:54:25.523595 26151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:25.525307 26136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:54:25.525612 26136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "ca3d5ce166f84170be37988787f69ed6"
format_stamp: "Formatted at 2026-06-19 18:54:25 on dist-test-slave-dm3f"
server_key: "475b8639d9a3c27f70ac6721a340d030"
server_key_iv: "d87af39ff76de5c44ba22d1b67a19cb0"
server_key_version: "encryptionkey@0"
I20260619 18:54:25.525892 26136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:25.561547 26136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:25.566833 26136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:25.567169 26136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:25.568280 26136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:25.569844 26136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:25.569976 26136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:25.570155 26136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:25.570232 26136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:25.607165 26136 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:45797
I20260619 18:54:25.607197 26263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:45797 every 8 connection(s)
I20260619 18:54:25.608692 26136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:54:25.617839 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 26136
I20260619 18:54:25.618116 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:54:25.618839 15724 external_mini_cluster.cc:1428] Setting key 6d71ac13f389e8555a864d0b896afa1a
Jun 19 18:54:25 dist-test-slave-dm3f krb5kdc[26030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895265, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:54:25.633774 26266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:25.611852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:36985 (local address 127.15.91.62:44771)
0619 18:54:25.612662 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:25.612778 (+ 116us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:25.613567 (+ 789us) server_negotiation.cc:407] Connection header received
0619 18:54:25.615820 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:25.615826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:25.615954 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:25.616096 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:25.617959 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:25.618936 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:25.621104 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:25.621548 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:25.621705 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:25.626166 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:25.626173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:25.626177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:25.626220 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:25.629224 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:25.630523 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:25.630528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:25.630531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:25.630779 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:25.631628 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:25.631632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:25.631635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:25.631982 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:25.632125 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:25.633254 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0619 18:54:25.633486 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":522,"threads_started":1}
I20260619 18:54:25.635536 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:25.612321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44771 (local address 127.15.91.1:36985)
0619 18:54:25.613331 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:25.613410 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:25.615426 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:25.616419 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:25.616430 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:25.617049 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:25.617709 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:25.617730 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:25.619152 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:25.619157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:25.620867 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:25.620890 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:25.621834 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:25.621842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:25.622138 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:25.623010 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:25.623043 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:25.625813 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:54:25.629468 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:54:25.629482 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:54:25.629505 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:54:25.630323 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:54:25.631003 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:54:25.631012 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:54:25.631019 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:54:25.631433 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:54:25.632178 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:54:25.632192 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:54:25.632743 (+ 551us) client_negotiation.cc:769] Sending connection context
0619 18:54:25.633122 (+ 379us) client_negotiation.cc:240] Negotiation successful
0619 18:54:25.633824 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":472,"threads_started":1}
I20260619 18:54:25.637616 26264 heartbeater.cc:347] Connected to a master server at 127.15.91.62:44771
I20260619 18:54:25.638118 26264 heartbeater.cc:464] Registering TS with master...
I20260619 18:54:25.639101 26264 heartbeater.cc:511] Master 127.15.91.62:44771 requested a full tablet report, sending...
I20260619 18:54:25.641880 26075 ts_manager.cc:194] Registered new tserver with Master: ca3d5ce166f84170be37988787f69ed6 (127.15.91.1:45797)
I20260619 18:54:25.643556 26075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:36985
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:54:25.679605 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44771
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:54:26.048590 26270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:26.049021 26270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:26.049226 26270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:54:26.058053 26270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:26.058214 26270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:26.058437 26270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:54:26.069231 26270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:26.070755 26270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:26.072218 26270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:26.088538 26275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:26.088508 26276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:26.088604 26278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:26.089943 26270 server_base.cc:1034] running on GCE node
I20260619 18:54:26.091204 26270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:26.092581 26270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:26.093914 26270 hybrid_clock.cc:648] HybridClock initialized: now 1781895266093855 us; error 77 us; skew 500 ppm
I20260619 18:54:26.094376 26270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:54:26 dist-test-slave-dm3f krb5kdc[26030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895266, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:54:26.098347 26270 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:54:26.100474 26270 webserver.cc:460] Webserver started at http://127.15.91.2:37349/ using document root <none> and password file <none>
I20260619 18:54:26.101341 26270 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:26.101526 26270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:26.101890 26270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:26.105455 26270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "c6a3c93168d942e4ac824547c39b9074"
format_stamp: "Formatted at 2026-06-19 18:54:26 on dist-test-slave-dm3f"
server_key: "2631c09aefcaca52a473fa691a205d71"
server_key_iv: "4a8c3388a6f304d734d819026a3218c0"
server_key_version: "encryptionkey@0"
I20260619 18:54:26.106444 26270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "c6a3c93168d942e4ac824547c39b9074"
format_stamp: "Formatted at 2026-06-19 18:54:26 on dist-test-slave-dm3f"
server_key: "2631c09aefcaca52a473fa691a205d71"
server_key_iv: "4a8c3388a6f304d734d819026a3218c0"
server_key_version: "encryptionkey@0"
I20260619 18:54:26.113178 26270 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:54:26.118026 26285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:26.119856 26270 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:54:26.120116 26270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "c6a3c93168d942e4ac824547c39b9074"
format_stamp: "Formatted at 2026-06-19 18:54:26 on dist-test-slave-dm3f"
server_key: "2631c09aefcaca52a473fa691a205d71"
server_key_iv: "4a8c3388a6f304d734d819026a3218c0"
server_key_version: "encryptionkey@0"
I20260619 18:54:26.120376 26270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:26.156749 26270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:26.161654 26270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:26.162001 26270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:26.163123 26270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:26.164779 26270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:26.164898 26270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:26.165089 26270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:26.165165 26270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:26.202674 26270 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:38973
I20260619 18:54:26.202777 26397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:38973 every 8 connection(s)
I20260619 18:54:26.204257 26270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:54:26.207154 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 26270
I20260619 18:54:26.207393 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:54:26.207851 15724 external_mini_cluster.cc:1428] Setting key 0c1beab0c5e0e0788e59d043300a775b
Jun 19 18:54:26 dist-test-slave-dm3f krb5kdc[26030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895266, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:54:26.230803 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.208501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:48035 (local address 127.15.91.62:44771)
0619 18:54:26.209097 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:26.209103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:26.210116 (+ 1013us) server_negotiation.cc:407] Connection header received
0619 18:54:26.212278 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:26.212284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:26.212393 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:26.212535 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:26.214317 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.215186 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.217623 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.218095 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.218256 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.222837 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:26.222843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:26.222847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:26.222896 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:26.226315 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:26.227681 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:26.227686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:26.227689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:26.227853 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:26.228511 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:26.228515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:26.228517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:26.228829 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:26.228989 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:26.230207 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0619 18:54:26.230447 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":306,"threads_started":1}
I20260619 18:54:26.232589 26401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.209052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44771 (local address 127.15.91.2:48035)
0619 18:54:26.209943 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:26.209990 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:26.211929 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:26.212842 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:26.212852 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:26.213435 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:26.214096 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.214116 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.215380 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.215385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.217394 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.217415 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.218379 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.218386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.218665 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.219524 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:26.219556 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:26.222399 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:54:26.226546 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:54:26.226560 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:54:26.226584 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:54:26.227470 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:54:26.228026 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:54:26.228035 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:54:26.228042 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:54:26.228363 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:54:26.228998 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:54:26.229011 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:54:26.229644 (+ 633us) client_negotiation.cc:769] Sending connection context
0619 18:54:26.230079 (+ 435us) client_negotiation.cc:240] Negotiation successful
0619 18:54:26.230873 (+ 794us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":396,"threads_started":1}
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
I20260619 18:54:26.234511 26398 heartbeater.cc:347] Connected to a master server at 127.15.91.62:44771
I20260619 18:54:26.234951 26398 heartbeater.cc:464] Registering TS with master...
I20260619 18:54:26.235831 26398 heartbeater.cc:511] Master 127.15.91.62:44771 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
I20260619 18:54:26.237746 26075 ts_manager.cc:194] Registered new tserver with Master: c6a3c93168d942e4ac824547c39b9074 (127.15.91.2:38973)
I20260619 18:54:26.239167 26075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:48035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:54:26.270085 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44771
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:54:26.643074 26404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:54:26.643478 26404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:54:26.643661 26404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 18:54:26.647470 26264 heartbeater.cc:503] Master 127.15.91.62:44771 was elected leader, sending a full tablet report...
W20260619 18:54:26.651625 26404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:54:26.651834 26404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:54:26.652036 26404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:54:26.662737 26404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:44771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:54:26.664206 26404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:54:26.665655 26404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:54:26.681074 26409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:26.680999 26410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:54:26.681097 26412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:54:26.681806 26404 server_base.cc:1034] running on GCE node
I20260619 18:54:26.682653 26404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:54:26.683980 26404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:54:26.685276 26404 hybrid_clock.cc:648] HybridClock initialized: now 1781895266685194 us; error 97 us; skew 500 ppm
I20260619 18:54:26.685796 26404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:54:26 dist-test-slave-dm3f krb5kdc[26030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895266, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:54:26.689476 26404 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:54:26.691361 26404 webserver.cc:460] Webserver started at http://127.15.91.3:33839/ using document root <none> and password file <none>
I20260619 18:54:26.692175 26404 fs_manager.cc:359] Metadata directory not provided
I20260619 18:54:26.692296 26404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:54:26.692564 26404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:54:26.696182 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "72b2986e5112415da81d8b7f4de771d3"
format_stamp: "Formatted at 2026-06-19 18:54:26 on dist-test-slave-dm3f"
server_key: "c671e6bd9a7eb3e83624d3bf02a66c85"
server_key_iv: "01dd1eeb4657d7ee711b920e42352e73"
server_key_version: "encryptionkey@0"
I20260619 18:54:26.697101 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "72b2986e5112415da81d8b7f4de771d3"
format_stamp: "Formatted at 2026-06-19 18:54:26 on dist-test-slave-dm3f"
server_key: "c671e6bd9a7eb3e83624d3bf02a66c85"
server_key_iv: "01dd1eeb4657d7ee711b920e42352e73"
server_key_version: "encryptionkey@0"
I20260619 18:54:26.703334 26404 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:54:26.707640 26419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:26.709398 26404 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:54:26.709652 26404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "72b2986e5112415da81d8b7f4de771d3"
format_stamp: "Formatted at 2026-06-19 18:54:26 on dist-test-slave-dm3f"
server_key: "c671e6bd9a7eb3e83624d3bf02a66c85"
server_key_iv: "01dd1eeb4657d7ee711b920e42352e73"
server_key_version: "encryptionkey@0"
I20260619 18:54:26.709936 26404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:54:26.728094 26404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:54:26.733451 26404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:54:26.733851 26404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:54:26.735059 26404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:54:26.736663 26404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:54:26.736749 26404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:26.736843 26404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:54:26.736894 26404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:54:26.774013 26404 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:45289
I20260619 18:54:26.774068 26531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:45289 every 8 connection(s)
I20260619 18:54:26.775590 26404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:54:26.777757 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 26404
I20260619 18:54:26.778012 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:54:26.778484 15724 external_mini_cluster.cc:1428] Setting key ec5bcc97b05499c21c0ef995288c46af
Jun 19 18:54:26 dist-test-slave-dm3f krb5kdc[26030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895266, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:54:26.800035 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.778952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:45141 (local address 127.15.91.62:44771)
0619 18:54:26.779808 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:26.779813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:26.780868 (+ 1055us) server_negotiation.cc:407] Connection header received
0619 18:54:26.783066 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:26.783072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:26.783160 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:26.783260 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:26.785298 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.786194 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.788143 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.788609 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.788749 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.792822 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:26.792828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:26.792831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:26.792872 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:26.796261 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:26.797348 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:26.797353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:26.797356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:26.797522 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:26.798057 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:26.798061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:26.798063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:26.798343 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:26.798487 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:26.799423 (+ 936us) server_negotiation.cc:299] Negotiation successful
0619 18:54:26.799668 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":600,"threads_started":1}
I20260619 18:54:26.801126 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.779662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:44771 (local address 127.15.91.3:45141)
0619 18:54:26.780689 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:26.780742 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:26.782645 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:26.783569 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:26.783579 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:26.784235 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:54:26.785038 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.785064 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.786407 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.786412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.787894 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.787914 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.788907 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.788913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.789151 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.789913 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:54:26.789939 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:54:26.792503 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:54:26.796538 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:54:26.796548 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:54:26.796575 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:54:26.797177 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:54:26.797670 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:54:26.797674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:54:26.797676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:54:26.797927 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:54:26.798499 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:54:26.798513 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:54:26.799022 (+ 509us) client_negotiation.cc:769] Sending connection context
0619 18:54:26.799295 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 18:54:26.799802 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":428,"threads_started":1}
I20260619 18:54:26.802954 26532 heartbeater.cc:347] Connected to a master server at 127.15.91.62:44771
I20260619 18:54:26.803330 26532 heartbeater.cc:464] Registering TS with master...
I20260619 18:54:26.804217 26532 heartbeater.cc:511] Master 127.15.91.62:44771 requested a full tablet report, sending...
I20260619 18:54:26.806397 26075 ts_manager.cc:194] Registered new tserver with Master: 72b2986e5112415da81d8b7f4de771d3 (127.15.91.3:45289)
I20260619 18:54:26.807617 26075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:45141
I20260619 18:54:26.808265 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:54:26.828442 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.813632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.15.91.62:44771)
0619 18:54:26.813823 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:26.813828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:26.814248 (+ 420us) server_negotiation.cc:407] Connection header received
0619 18:54:26.814438 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:26.814441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:26.814512 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:26.814631 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:26.816354 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.817128 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.818547 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.819107 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.819239 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.821874 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:26.821880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:26.821883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:26.821929 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:26.824788 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:26.825769 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:26.825774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:26.825776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:26.825934 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:26.826823 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:26.826844 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:26.826851 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:26.827265 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:26.827536 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:26.827962 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 18:54:26.828169 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:54:26.833388 26075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:54:26.841403 26075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:54:26.870538 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.857667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.15.91.1:45797)
0619 18:54:26.858356 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:26.858366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:26.858953 (+ 587us) server_negotiation.cc:407] Connection header received
0619 18:54:26.859091 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:26.859104 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:26.859460 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:26.859670 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:54:26.860681 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.863997 (+ 3316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.867269 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.869040 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.869230 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.869528 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:26.869939 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 18:54:26.870140 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":730,"threads_started":1}
I20260619 18:54:26.870653 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.857461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:45797 (local address 127.0.0.1:39984)
0619 18:54:26.858365 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:26.858417 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:26.858630 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:26.859832 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:26.859839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:26.859872 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:54:26.860324 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.860342 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.864301 (+ 3959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.864311 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.866981 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.866997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.869302 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.869310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.869611 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.869698 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 18:54:26.869889 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 18:54:26.870129 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":26,"thread_start_us":353,"threads_started":1}
I20260619 18:54:26.874552 26547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.858151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:45289 (local address 127.0.0.1:48162)
0619 18:54:26.859052 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:26.859072 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:26.859179 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:26.865541 (+ 6362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:26.865548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:26.865587 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:54:26.866106 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.866120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.868581 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.868588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.871413 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.871425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.873547 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.873555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.873747 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.873789 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 18:54:26.873908 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 18:54:26.874113 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":4,"thread_start_us":979,"threads_started":1}
I20260619 18:54:26.874766 26549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.858864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.15.91.3:45289)
0619 18:54:26.864739 (+ 5875us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:26.864749 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:26.864790 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:54:26.864920 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:26.864929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:26.865250 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:26.865523 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:54:26.866921 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.868406 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.871633 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.873256 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.873424 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.873691 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:26.874169 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 18:54:26.874358 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5671,"thread_start_us":5472,"threads_started":1}
I20260619 18:54:26.875439 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.858824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38973 (local address 127.0.0.1:48864)
0619 18:54:26.865538 (+ 6714us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:26.865558 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:26.865727 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:26.866555 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:26.866559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:26.866591 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:54:26.867103 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.867117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.869085 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.869090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.871389 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.871412 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.873859 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.873868 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.874686 (+ 818us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.874768 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 18:54:26.874872 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 18:54:26.875040 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6530,"mutex_wait_us":567,"spinlock_wait_cycles":8576,"thread_start_us":6985,"threads_started":1}
I20260619 18:54:26.875865 26550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.860655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.15.91.2:38973)
0619 18:54:26.865151 (+ 4496us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:26.865162 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:26.865901 (+ 739us) server_negotiation.cc:407] Connection header received
0619 18:54:26.866039 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:26.866048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:26.866329 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:26.866550 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:54:26.867327 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.868906 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.871632 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.873256 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.873424 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.873662 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:26.875340 (+ 1678us) server_negotiation.cc:299] Negotiation successful
0619 18:54:26.875539 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4321,"thread_start_us":4164,"threads_started":1}
I20260619 18:54:26.875630 26199 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcad83d8d4a4072b7271ebb3ff9cda2 (DEFAULT_TABLE table=test-table [id=3b61228a9fab45e3ba3a259ac7b9fe7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:54:26.878041 26199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcad83d8d4a4072b7271ebb3ff9cda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:54:26.879455 26467 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcad83d8d4a4072b7271ebb3ff9cda2 (DEFAULT_TABLE table=test-table [id=3b61228a9fab45e3ba3a259ac7b9fe7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:54:26.880162 26333 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcad83d8d4a4072b7271ebb3ff9cda2 (DEFAULT_TABLE table=test-table [id=3b61228a9fab45e3ba3a259ac7b9fe7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:54:26.881256 26467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcad83d8d4a4072b7271ebb3ff9cda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:54:26.881616 26333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcad83d8d4a4072b7271ebb3ff9cda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:54:26.894616 26551 tablet_bootstrap.cc:492] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6: Bootstrap starting.
I20260619 18:54:26.898413 26551 tablet_bootstrap.cc:654] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6: Neither blocks nor log segments found. Creating new log.
I20260619 18:54:26.899556 26553 tablet_bootstrap.cc:492] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: Bootstrap starting.
I20260619 18:54:26.900112 26551 log.cc:824] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6: Log is configured to *not* fsync() on all Append() calls
I20260619 18:54:26.903400 26552 tablet_bootstrap.cc:492] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: Bootstrap starting.
I20260619 18:54:26.905047 26553 tablet_bootstrap.cc:654] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: Neither blocks nor log segments found. Creating new log.
I20260619 18:54:26.906566 26553 log.cc:824] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:54:26.907418 26552 tablet_bootstrap.cc:654] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: Neither blocks nor log segments found. Creating new log.
I20260619 18:54:26.908304 26551 tablet_bootstrap.cc:492] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6: No bootstrap required, opened a new log
I20260619 18:54:26.908704 26551 ts_tablet_manager.cc:1397] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260619 18:54:26.908967 26552 log.cc:824] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: Log is configured to *not* fsync() on all Append() calls
I20260619 18:54:26.910969 26553 tablet_bootstrap.cc:492] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: No bootstrap required, opened a new log
I20260619 18:54:26.911306 26553 ts_tablet_manager.cc:1397] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20260619 18:54:26.914237 26552 tablet_bootstrap.cc:492] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: No bootstrap required, opened a new log
I20260619 18:54:26.915084 26552 ts_tablet_manager.cc:1397] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260619 18:54:26.921392 26551 raft_consensus.cc:348] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3d5ce166f84170be37988787f69ed6" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45797 } } peers { permanent_uuid: "72b2986e5112415da81d8b7f4de771d3" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45289 } } peers { permanent_uuid: "c6a3c93168d942e4ac824547c39b9074" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38973 } }
I20260619 18:54:26.922091 26551 raft_consensus.cc:374] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:54:26.922242 26551 raft_consensus.cc:729] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3d5ce166f84170be37988787f69ed6, State: Initialized, Role: FOLLOWER
I20260619 18:54:26.923123 26551 consensus_queue.cc:260] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3d5ce166f84170be37988787f69ed6" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45797 } } peers { permanent_uuid: "72b2986e5112415da81d8b7f4de771d3" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45289 } } peers { permanent_uuid: "c6a3c93168d942e4ac824547c39b9074" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38973 } }
I20260619 18:54:26.924557 26553 raft_consensus.cc:348] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3d5ce166f84170be37988787f69ed6" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45797 } } peers { permanent_uuid: "72b2986e5112415da81d8b7f4de771d3" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45289 } } peers { permanent_uuid: "c6a3c93168d942e4ac824547c39b9074" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38973 } }
I20260619 18:54:26.925956 26553 raft_consensus.cc:374] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:54:26.925993 26551 ts_tablet_manager.cc:1428] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260619 18:54:26.926086 26552 raft_consensus.cc:348] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3d5ce166f84170be37988787f69ed6" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45797 } } peers { permanent_uuid: "72b2986e5112415da81d8b7f4de771d3" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45289 } } peers { permanent_uuid: "c6a3c93168d942e4ac824547c39b9074" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38973 } }
I20260619 18:54:26.926672 26552 raft_consensus.cc:374] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:54:26.926895 26552 raft_consensus.cc:729] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a3c93168d942e4ac824547c39b9074, State: Initialized, Role: FOLLOWER
I20260619 18:54:26.927115 26553 raft_consensus.cc:729] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b2986e5112415da81d8b7f4de771d3, State: Initialized, Role: FOLLOWER
I20260619 18:54:26.928121 26553 consensus_queue.cc:260] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3d5ce166f84170be37988787f69ed6" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45797 } } peers { permanent_uuid: "72b2986e5112415da81d8b7f4de771d3" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45289 } } peers { permanent_uuid: "c6a3c93168d942e4ac824547c39b9074" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38973 } }
I20260619 18:54:26.928874 26552 consensus_queue.cc:260] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3d5ce166f84170be37988787f69ed6" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45797 } } peers { permanent_uuid: "72b2986e5112415da81d8b7f4de771d3" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45289 } } peers { permanent_uuid: "c6a3c93168d942e4ac824547c39b9074" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38973 } }
I20260619 18:54:26.931668 26532 heartbeater.cc:503] Master 127.15.91.62:44771 was elected leader, sending a full tablet report...
I20260619 18:54:26.932034 26552 ts_tablet_manager.cc:1428] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
I20260619 18:54:26.932235 26553 ts_tablet_manager.cc:1428] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: Time spent starting tablet: real 0.021s user 0.013s sys 0.005s
I20260619 18:54:26.933004 26398 heartbeater.cc:503] Master 127.15.91.62:44771 was elected leader, sending a full tablet report...
W20260619 18:54:26.957333 26399 tablet.cc:2369] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:54:26.987317 26558 raft_consensus.cc:482] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:54:26.987726 26558 raft_consensus.cc:504] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3d5ce166f84170be37988787f69ed6" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45797 } } peers { permanent_uuid: "72b2986e5112415da81d8b7f4de771d3" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45289 } } peers { permanent_uuid: "c6a3c93168d942e4ac824547c39b9074" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38973 } }
I20260619 18:54:26.989647 26558 leader_election.cc:288] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca3d5ce166f84170be37988787f69ed6 (127.15.91.1:45797), c6a3c93168d942e4ac824547c39b9074 (127.15.91.2:38973)
I20260619 18:54:26.996889 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.990105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:45797 (local address 127.15.91.3:60435)
0619 18:54:26.990349 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:26.990367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:26.990523 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:26.991129 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:26.991135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:26.991171 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:54:26.991337 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.991345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.992801 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.992805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.994701 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.994783 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.996257 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.996261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.996426 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.996454 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:54:26.996540 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 18:54:26.996650 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 18:54:26.997026 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.990210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:60435 (local address 127.15.91.1:45797)
0619 18:54:26.990472 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:26.990479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:26.990501 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:54:26.990780 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:26.990789 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:26.990884 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:26.990985 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:54:26.991505 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.992640 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.994981 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.996049 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.996223 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.996360 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:26.996601 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 18:54:26.996755 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:54:26.998111 26550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.990785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:35953 (local address 127.15.91.2:38973)
0619 18:54:26.991042 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:26.991047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:26.991724 (+ 677us) server_negotiation.cc:407] Connection header received
0619 18:54:26.991896 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:26.991903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:26.991999 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:26.992096 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:54:26.992639 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.993399 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.995671 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.997037 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.997200 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.997309 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:26.997781 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 18:54:26.997910 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:54:26.998266 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:26.990559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38973 (local address 127.15.91.3:35953)
0619 18:54:26.991576 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0619 18:54:26.991594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:54:26.991757 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:54:26.992123 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:54:26.992126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:54:26.992146 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:54:26.992462 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.992470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.993592 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.993598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.995465 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:54:26.995480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:54:26.997443 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:54:26.997448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:54:26.997579 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:26.997604 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:54:26.997741 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 18:54:26.997909 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":566,"threads_started":1}
I20260619 18:54:26.998291 26219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcad83d8d4a4072b7271ebb3ff9cda2" candidate_uuid: "72b2986e5112415da81d8b7f4de771d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3d5ce166f84170be37988787f69ed6" is_pre_election: true
I20260619 18:54:26.998998 26219 raft_consensus.cc:2454] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b2986e5112415da81d8b7f4de771d3 in term 0.
I20260619 18:54:26.999522 26353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcad83d8d4a4072b7271ebb3ff9cda2" candidate_uuid: "72b2986e5112415da81d8b7f4de771d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3c93168d942e4ac824547c39b9074" is_pre_election: true
I20260619 18:54:26.999821 26420 leader_election.cc:302] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72b2986e5112415da81d8b7f4de771d3, ca3d5ce166f84170be37988787f69ed6; no voters:
I20260619 18:54:27.000108 26353 raft_consensus.cc:2454] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b2986e5112415da81d8b7f4de771d3 in term 0.
I20260619 18:54:27.000267 26558 raft_consensus.cc:2781] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:54:27.000459 26558 raft_consensus.cc:482] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:54:27.000597 26558 raft_consensus.cc:3037] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:54:27.003173 26558 raft_consensus.cc:504] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3d5ce166f84170be37988787f69ed6" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45797 } } peers { permanent_uuid: "72b2986e5112415da81d8b7f4de771d3" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45289 } } peers { permanent_uuid: "c6a3c93168d942e4ac824547c39b9074" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38973 } }
I20260619 18:54:27.003880 26558 leader_election.cc:288] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [CANDIDATE]: Term 1 election: Requested vote from peers ca3d5ce166f84170be37988787f69ed6 (127.15.91.1:45797), c6a3c93168d942e4ac824547c39b9074 (127.15.91.2:38973)
I20260619 18:54:27.004606 26219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcad83d8d4a4072b7271ebb3ff9cda2" candidate_uuid: "72b2986e5112415da81d8b7f4de771d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3d5ce166f84170be37988787f69ed6"
I20260619 18:54:27.004681 26353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcad83d8d4a4072b7271ebb3ff9cda2" candidate_uuid: "72b2986e5112415da81d8b7f4de771d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3c93168d942e4ac824547c39b9074"
I20260619 18:54:27.004899 26219 raft_consensus.cc:3037] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:54:27.004961 26353 raft_consensus.cc:3037] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:54:27.007633 26219 raft_consensus.cc:2454] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b2986e5112415da81d8b7f4de771d3 in term 1.
I20260619 18:54:27.007637 26353 raft_consensus.cc:2454] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b2986e5112415da81d8b7f4de771d3 in term 1.
I20260619 18:54:27.008322 26420 leader_election.cc:302] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72b2986e5112415da81d8b7f4de771d3, c6a3c93168d942e4ac824547c39b9074; no voters:
I20260619 18:54:27.008729 26558 raft_consensus.cc:2781] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:54:27.009276 26558 raft_consensus.cc:686] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 1 LEADER]: Becoming Leader. State: Replica: 72b2986e5112415da81d8b7f4de771d3, State: Running, Role: LEADER
I20260619 18:54:27.009835 26558 consensus_queue.cc:237] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3d5ce166f84170be37988787f69ed6" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45797 } } peers { permanent_uuid: "72b2986e5112415da81d8b7f4de771d3" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45289 } } peers { permanent_uuid: "c6a3c93168d942e4ac824547c39b9074" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38973 } }
I20260619 18:54:27.015679 26074 catalog_manager.cc:5495] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72b2986e5112415da81d8b7f4de771d3 (127.15.91.3). New cstate: current_term: 1 leader_uuid: "72b2986e5112415da81d8b7f4de771d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3d5ce166f84170be37988787f69ed6" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b2986e5112415da81d8b7f4de771d3" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6a3c93168d942e4ac824547c39b9074" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38973 } health_report { overall_health: UNKNOWN } } }
W20260619 18:54:27.027547 26533 tablet.cc:2369] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:54:27.072785 26549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:27.066402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.15.91.3:45289)
0619 18:54:27.066684 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:27.066690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:27.066745 (+ 55us) server_negotiation.cc:407] Connection header received
0619 18:54:27.066892 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:27.066896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:27.066990 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:27.067140 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:54:27.067688 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:27.068460 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:27.070007 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:27.070439 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:27.070596 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:27.071465 (+ 869us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:54:27.072037 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:54:27.072167 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:27.072357 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 18:54:27.072440 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 18:54:27.083187 26219 raft_consensus.cc:1261] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6 [term 1 FOLLOWER]: Refusing update from remote peer 72b2986e5112415da81d8b7f4de771d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:54:27.083187 26353 raft_consensus.cc:1261] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074 [term 1 FOLLOWER]: Refusing update from remote peer 72b2986e5112415da81d8b7f4de771d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:54:27.084578 26558 consensus_queue.cc:1035] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3d5ce166f84170be37988787f69ed6" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 45797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:54:27.085057 26561 consensus_queue.cc:1035] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6a3c93168d942e4ac824547c39b9074" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:54:27.099640 26564 mvcc.cc:205] Tried to move back new op lower bound from 7298643013957664768 to 7298643013677236224. Current Snapshot: MvccSnapshot[applied={T|T < 7298643013957664768}]
I20260619 18:54:27.120452 26075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:54:27.120836 26075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:54:27.125598 26075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb446b2bb69644af8f5699c722d16cc8: Sending DeleteTablet for 3 replicas of tablet 8fcad83d8d4a4072b7271ebb3ff9cda2
I20260619 18:54:27.127244 26333 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcad83d8d4a4072b7271ebb3ff9cda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:54:27 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:48864
I20260619 18:54:27.127522 26199 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcad83d8d4a4072b7271ebb3ff9cda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:54:27 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:39984
I20260619 18:54:27.128116 26467 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcad83d8d4a4072b7271ebb3ff9cda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:54:27 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:48162
I20260619 18:54:27.129323 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 26136
I20260619 18:54:27.136590 26575 tablet_replica.cc:330] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6: stopping tablet replica
I20260619 18:54:27.137326 26576 tablet_replica.cc:330] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: stopping tablet replica
I20260619 18:54:27.138083 26576 raft_consensus.cc:2229] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:54:27.138689 26575 raft_consensus.cc:2229] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:54:27.139465 26575 raft_consensus.cc:2258] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:54:27.140353 26576 raft_consensus.cc:2258] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:54:27.142635 26574 tablet_replica.cc:330] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: stopping tablet replica
I20260619 18:54:27.143144 26574 raft_consensus.cc:2229] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:54:27.143321 26575 ts_tablet_manager.cc:1905] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:54:27.143748 26574 raft_consensus.cc:2258] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:54:27.148057 26574 ts_tablet_manager.cc:1905] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:54:27.148057 26576 ts_tablet_manager.cc:1905] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:54:27 dist-test-slave-dm3f krb5kdc[26030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:54:27.156040 26575 ts_tablet_manager.cc:1918] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:54:27.156453 26575 log.cc:1196] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/8fcad83d8d4a4072b7271ebb3ff9cda2
I20260619 18:54:27.156828 26576 ts_tablet_manager.cc:1918] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:54:27.157164 26576 log.cc:1196] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/8fcad83d8d4a4072b7271ebb3ff9cda2
I20260619 18:54:27.157805 26575 ts_tablet_manager.cc:1939] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P ca3d5ce166f84170be37988787f69ed6: Deleting consensus metadata
I20260619 18:54:27.157888 26576 ts_tablet_manager.cc:1939] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P 72b2986e5112415da81d8b7f4de771d3: Deleting consensus metadata
I20260619 18:54:27.160127 26060 catalog_manager.cc:4841] TS 72b2986e5112415da81d8b7f4de771d3 (127.15.91.3:45289): tablet 8fcad83d8d4a4072b7271ebb3ff9cda2 (table test-table [id=3b61228a9fab45e3ba3a259ac7b9fe7c]) successfully deleted
I20260619 18:54:27.161559 26059 catalog_manager.cc:4841] TS ca3d5ce166f84170be37988787f69ed6 (127.15.91.1:45797): tablet 8fcad83d8d4a4072b7271ebb3ff9cda2 (table test-table [id=3b61228a9fab45e3ba3a259ac7b9fe7c]) successfully deleted
I20260619 18:54:27.164840 26574 ts_tablet_manager.cc:1918] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:54:27.165071 26574 log.cc:1196] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/8fcad83d8d4a4072b7271ebb3ff9cda2
I20260619 18:54:27.165587 26574 ts_tablet_manager.cc:1939] T 8fcad83d8d4a4072b7271ebb3ff9cda2 P c6a3c93168d942e4ac824547c39b9074: Deleting consensus metadata
I20260619 18:54:27.167660 26059 catalog_manager.cc:4841] TS c6a3c93168d942e4ac824547c39b9074 (127.15.91.2:38973): tablet 8fcad83d8d4a4072b7271ebb3ff9cda2 (table test-table [id=3b61228a9fab45e3ba3a259ac7b9fe7c]) successfully deleted
I20260619 18:54:27.168200 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:27.131597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39998 (local address 127.15.91.1:45797)
0619 18:54:27.131861 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:27.131873 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:27.136210 (+ 4337us) server_negotiation.cc:407] Connection header received
0619 18:54:27.136720 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:27.136729 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:27.136842 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:27.136937 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:27.138963 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:27.140144 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:27.144255 (+ 4111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:27.144979 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:27.145113 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:27.159042 (+ 13929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:27.159065 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:27.159087 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:27.159149 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:27.163240 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:27.164412 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:27.164426 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:27.164432 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:27.164630 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:27.165199 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:27.165206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:27.165210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:27.166194 (+ 984us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:27.166791 (+ 597us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:27.167197 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 18:54:27.167546 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:54:27.169051 26259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:27.277902 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 26136
I20260619 18:54:27.303438 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 26270
Jun 19 18:54:27 dist-test-slave-dm3f krb5kdc[26030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:54:27.325768 26550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:27.304682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.15.91.2:38973)
0619 18:54:27.304909 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:27.304914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:27.304939 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:54:27.305083 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:27.305088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:27.305176 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:27.305290 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:27.307171 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:27.308050 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:27.310136 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:27.310582 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:27.310786 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:27.316540 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:27.316562 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:27.316586 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:27.316645 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:27.321117 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:27.322139 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:27.322151 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:27.322155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:27.322500 (+ 345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:27.323245 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:27.323252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:27.323256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:27.324194 (+ 938us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:27.324593 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:27.325140 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 18:54:27.325427 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:54:27.326678 26393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:27.427840 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 26270
I20260619 18:54:27.448462 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 26404
Jun 19 18:54:27 dist-test-slave-dm3f krb5kdc[26030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:54:27.469720 26549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:54:27.449399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.15.91.3:45289)
0619 18:54:27.449579 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 18:54:27.449584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:54:27.449611 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:54:27.449832 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:54:27.449837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:54:27.449910 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:54:27.449998 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:54:27.451966 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:27.452802 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:27.454866 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:54:27.455386 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:54:27.455520 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:54:27.462181 (+ 6661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:54:27.462200 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:54:27.462217 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:54:27.462271 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:54:27.466054 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:27.467163 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:27.467172 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:27.467174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:27.467367 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:54:27.467840 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:54:27.467852 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:54:27.467855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:54:27.468308 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:54:27.468708 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0619 18:54:27.469027 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 18:54:27.469288 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:54:27.470650 26527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:27.600889 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 26404
I20260619 18:54:27.621547 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 26043
I20260619 18:54:27.622941 26105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:54:27.740410 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 26043
2026-06-19T18:54:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4694 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:54:27 dist-test-slave-dm3f krb5kdc[26584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:54:27 dist-test-slave-dm3f krb5kdc[26584](info): set up 2 sockets
Jun 19 18:54:27 dist-test-slave-dm3f krb5kdc[26584](info): commencing operation
krb5kdc: starting...
W20260619 18:54:28.872750 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:54:29 dist-test-slave-dm3f krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:54:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.91.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.91.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/rangeradmin_127.15.91.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/rangeradmin_127.15.91.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.91.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.91.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/rangerlookup_127.15.91.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/rangerlookup_127.15.91.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.91.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.91.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/HTTP_127.15.91.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/HTTP_127.15.91.20@KRBTEST.COM.keytab.
I20260619 18:54:29.246217 15724 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskualRwQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/postgres -l logfile start
2026-06-19 18:54:31.910 UTC [26614] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 18:54:31.911 UTC [26614] LOG: listening on IPv4 address "127.15.91.20", port 58383
2026-06-19 18:54:31.914 UTC [26614] LOG: listening on Unix socket "/tmp/.s.PGSQL.58383"
2026-06-19 18:54:31.938 UTC [26617] LOG: database system was shut down at 2026-06-19 18:54:30 UTC
2026-06-19 18:54:31.942 UTC [26614] LOG: database system is ready to accept connections
I20260619 18:54:32.927497 15724 mini_postgres.cc:96] Postgres bound to 58383
2026-06-19 18:54:32.946 UTC [26625] FATAL: database "slave" does not exist
127.15.91.20:58383 - accepting connections
I20260619 18:54:32.946964 15724 mini_ranger.cc:161] Starting Ranger...
I20260619 18:54:32.977610 15724 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 18:54:33.515498 15724 mini_ranger.cc:87] Created ranger Postgres database
I20260619 18:54:33.515623 15724 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-admin
2026-06-19 18:54:33,840 [I] DB FLAVOR :POSTGRES
2026-06-19 18:54:33,841 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:54:33,841 [I] Checking connection..
2026-06-19 18:54:33,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 18:54:34,108 [I] Checking connection passed.
2026-06-19 18:54:34,109 [I] --------- Verifying version history table ---------
2026-06-19 18:54:34,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:54:34,391 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:54:34,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:54:34,685 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:54:34,685 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 18:54:34,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 18:54:34.967 UTC [26689] WARNING: there is no transaction in progress
2026-06-19 18:54:34,975 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 18:54:34,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:54:35,260 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 18:54:35,260 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:54:35,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 18:54:35,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 18:54:35,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dm3f.c.gcp-upstream.internal','N') ;"
2026-06-19 18:54:36,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:54:36,337 [I] Table x_portal_user does not exist in database ranger
2026-06-19 18:54:36,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:54:36,611 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 18:54:36,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:54:36,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:54:37,147 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 18:54:37,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 18:54:38.266 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.284 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.303 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.322 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.339 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.360 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.426 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.434 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.451 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.464 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.478 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.494 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.506 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.514 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.522 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:38.530 UTC [26797] WARNING: there is no transaction in progress
2026-06-19 18:54:39,331 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 18:54:39,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 18:54:39,585 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 18:54:39,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:54:39,861 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 18:54:39,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:54:40,151 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 18:54:40,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:54:40,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:54:40,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dm3f.c.gcp-upstream.internal';"
2026-06-19 18:54:40,937 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 18:54:40,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 18:54:41,197 [I] DB_PATCHES have already been applied
I20260619 18:54:41.203660 15724 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-admin:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskualRwQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 18:54:41.203792 15724 mini_ranger.cc:193] Using host: 127.15.91.20
I20260619 18:54:41.219703 15724 mini_ranger.cc:227] Ranger admin URL: http://127.15.91.20:35975
Jun 19, 2026 6:54:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:54:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:54:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:54:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:54:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:54:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:54:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 18:54:42.394235 15724 mini_ranger.cc:160] Time spent starting Ranger: real 9.447s user 0.000s sys 0.046s
Jun 19, 2026 6:55:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:55:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 6:55:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:55:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:55:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 18:55:07.682850 15724 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.91.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.91.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/rangerkms_127.15.91.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/rangerkms_127.15.91.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.91.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.91.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.91.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/HTTP_127.15.91.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.91.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/HTTP_127.15.91.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 18:55:07 dist-test-slave-dm3f krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895307, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 18:55:07.861447 15724 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 18:55:07.886054 15724 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 18:55:08.849622 15724 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 18:55:08.849808 15724 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-kms
I20260619 18:55:08.849931 15724 mini_ranger_kms.cc:227] Using postgres at 127.15.91.20:58383
2026-06-19 18:55:09,323 [I] DB FLAVOR :POSTGRES
2026-06-19 18:55:09,324 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:55:09,324 [I] Checking connection
2026-06-19 18:55:09,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 18:55:09,620 [I] connection success
2026-06-19 18:55:09,620 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 18:55:09,621 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 18:55:09,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 18:55:09,922 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 18:55:09,923 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:55:09,923 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 18:55:09,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:58383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 18:55:10,250 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 18:55:10.893657 15724 mini_ranger_kms.cc:310] Created kms service
I20260619 18:55:11.088119 15724 mini_ranger_kms.cc:326] Created kudu user
I20260619 18:55:11.140379 15724 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 18:55:11 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895307, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:55:11.582882 15724 mini_ranger_kms.cc:384] Added ranger policy
I20260619 18:55:11.583233 15724 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-kms:/tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 18:55:11.583289 15724 mini_ranger_kms.cc:240] Using host: 127.15.91.20
I20260619 18:55:11.604928 15724 mini_ranger_kms.cc:276] Ranger KMS PID: 26978
I20260619 18:55:11.605150 15724 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.91.20:51341
Jun 19, 2026 6:55:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:55:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:55:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:55:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:55:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 18:55:13.972174 15724 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.111s user 0.115s sys 0.382s
I20260619 18:55:13.972461 15724 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 18:55:13.972688 15724 mini_ranger_kms.cc:401] 127.15.91.20:51341/kms/v1/keys
Jun 19 18:55:14 dist-test-slave-dm3f krb5kdc[26584](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781895314, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.91.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 18:55:14 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895314, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.91.20@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
Jun 19 18:55:14 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895314, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.91.20@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
Jun 19 18:55:17 dist-test-slave-dm3f krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:55:17.530822 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:39087
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:39087
--ranger_config_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:51341/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:55:17.913302 27028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:55:17.913703 27028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:55:17.913868 27028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:55:17.921689 27028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:55:17.921887 27028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:55:17.922003 27028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:55:17.922107 27028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:55:17.922190 27028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:55:17.933244 27028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.91.20:51341/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:39087
--ranger_config_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:39087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:55:17.934863 27028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:55:17.936591 27028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:55:17.951076 27033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:55:17.950853 27036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:55:17.951135 27034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:55:17.952069 27028 server_base.cc:1034] running on GCE node
I20260619 18:55:17.952929 27028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:55:17.954901 27028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:55:17.956183 27028 hybrid_clock.cc:648] HybridClock initialized: now 1781895317956124 us; error 49 us; skew 500 ppm
I20260619 18:55:17.956677 27028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:55:17 dist-test-slave-dm3f krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895317, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:55:17.963567 27028 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:55:17.965420 27028 webserver.cc:460] Webserver started at http://127.15.91.62:38259/ using document root <none> and password file <none>
I20260619 18:55:17.966216 27028 fs_manager.cc:359] Metadata directory not provided
I20260619 18:55:17.966353 27028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:55:17.966641 27028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:55:17 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895317, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:55:18.040355 27028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "0d77e28316cc4f1dabbf0c982909a756"
format_stamp: "Formatted at 2026-06-19 18:55:18 on dist-test-slave-dm3f"
server_key: "84e131808c4d102ca6b20cafe0c49c6f"
server_key_iv: "e28dd2acf4a933caade4362480b3fe8b"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:18.041376 27028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "0d77e28316cc4f1dabbf0c982909a756"
format_stamp: "Formatted at 2026-06-19 18:55:18 on dist-test-slave-dm3f"
server_key: "84e131808c4d102ca6b20cafe0c49c6f"
server_key_iv: "e28dd2acf4a933caade4362480b3fe8b"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:18.047672 27028 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 18:55:18.069365 27043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:55:18.071213 27028 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:55:18.071445 27028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "0d77e28316cc4f1dabbf0c982909a756"
format_stamp: "Formatted at 2026-06-19 18:55:18 on dist-test-slave-dm3f"
server_key: "84e131808c4d102ca6b20cafe0c49c6f"
server_key_iv: "e28dd2acf4a933caade4362480b3fe8b"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:18.071707 27028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:55:18.090596 27028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:55:18.095537 27028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:55:18.095909 27028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:55:18.118206 27028 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:39087
I20260619 18:55:18.118239 27094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:39087 every 8 connection(s)
I20260619 18:55:18.119886 27028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:55:18.124665 27095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:55:18.124825 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 27028
I20260619 18:55:18.125103 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:55:18.125811 15724 external_mini_cluster.cc:1428] Setting key aecb1baaa6673a068c982685caeeb645
I20260619 18:55:18.141386 27095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756: Bootstrap starting.
Jun 19 18:55:18 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:55:18.145850 27095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756: Neither blocks nor log segments found. Creating new log.
I20260619 18:55:18.147318 27095 log.cc:824] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756: Log is configured to *not* fsync() on all Append() calls
I20260619 18:55:18.151363 27095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756: No bootstrap required, opened a new log
I20260619 18:55:18.155117 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:18.128189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.15.91.62:39087)
0619 18:55:18.129183 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:18.129198 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:18.129245 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:55:18.131469 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:18.131505 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:18.132085 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:18.132507 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:55:18.134331 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:18.135970 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:18.138068 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:18.139544 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:18.139833 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:18.145319 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:55:18.145329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:55:18.145349 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:55:18.145404 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:55:18.149497 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:18.150454 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:18.150464 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:18.150474 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:18.150749 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:18.151206 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:18.151211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:18.151214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:18.152078 (+ 864us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:55:18.152443 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:18.152978 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 18:55:18.153487 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":449,"threads_started":1}
I20260619 18:55:18.161095 27095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d77e28316cc4f1dabbf0c982909a756" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39087 } }
I20260619 18:55:18.161547 27095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:55:18.161631 27095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77e28316cc4f1dabbf0c982909a756, State: Initialized, Role: FOLLOWER
I20260619 18:55:18.162245 27095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d77e28316cc4f1dabbf0c982909a756" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39087 } }
I20260619 18:55:18.162503 27095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:55:18.162614 27095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:55:18.162871 27095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:55:18.165604 27095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d77e28316cc4f1dabbf0c982909a756" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39087 } }
I20260619 18:55:18.166373 27095 leader_election.cc:302] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d77e28316cc4f1dabbf0c982909a756; no voters:
I20260619 18:55:18.167071 27095 leader_election.cc:288] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:55:18.167232 27100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:55:18.167995 27100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77e28316cc4f1dabbf0c982909a756, State: Running, Role: LEADER
I20260619 18:55:18.168606 27100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d77e28316cc4f1dabbf0c982909a756" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39087 } }
I20260619 18:55:18.169626 27095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:55:18.173727 27102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d77e28316cc4f1dabbf0c982909a756. Latest consensus state: current_term: 1 leader_uuid: "0d77e28316cc4f1dabbf0c982909a756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d77e28316cc4f1dabbf0c982909a756" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39087 } } }
I20260619 18:55:18.174293 27102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [sys.catalog]: This master's current role is: LEADER
I20260619 18:55:18.175042 27101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d77e28316cc4f1dabbf0c982909a756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d77e28316cc4f1dabbf0c982909a756" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39087 } } }
I20260619 18:55:18.175408 27101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756 [sys.catalog]: This master's current role is: LEADER
I20260619 18:55:18.180125 27108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:55:18.186743 27108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:55:18.187657 27095 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:55:18.207952 27108 catalog_manager.cc:1347] Generated new cluster ID: eda3396d9b4e43fe89a010118378e8cc
I20260619 18:55:18.208134 27108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:55:18.242097 27108 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:55:18.244347 27108 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:55:18.253508 27108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756: Generated new TSK 0
I20260619 18:55:18.254889 27108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:55:19.070885 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:51341/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39087
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:55:19 dist-test-slave-dm3f krb5kdc[26584](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781895319, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 18:55:19.583159 27140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:55:19.583637 27140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:55:19.583833 27140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:55:19.597023 27140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:55:19.597215 27140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:55:19.597440 27140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:55:19.617053 27140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.91.20:51341/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:55:19.619212 27140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:55:19.621488 27140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:55:19.643000 27150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:55:19.650557 27151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:55:19.650671 27153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:55:19.652072 27140 server_base.cc:1034] running on GCE node
I20260619 18:55:19.653292 27140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:55:19.654896 27140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:55:19.657388 27140 hybrid_clock.cc:648] HybridClock initialized: now 1781895319657312 us; error 62 us; skew 500 ppm
I20260619 18:55:19.657943 27140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:55:19 dist-test-slave-dm3f krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895319, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:55:19.663205 27140 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:55:19.671180 27140 webserver.cc:460] Webserver started at http://127.15.91.1:43355/ using document root <none> and password file <none>
I20260619 18:55:19.672375 27140 fs_manager.cc:359] Metadata directory not provided
I20260619 18:55:19.672577 27140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:55:19.673043 27140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:55:19 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895319, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:55:19.699286 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "97392ce8c68d4457acc58f3898677033"
format_stamp: "Formatted at 2026-06-19 18:55:19 on dist-test-slave-dm3f"
server_key: "0830197cbb0e241734b90fa71d8f0fd4"
server_key_iv: "47d046b3745714b990953956d30b0614"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:19.700731 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "97392ce8c68d4457acc58f3898677033"
format_stamp: "Formatted at 2026-06-19 18:55:19 on dist-test-slave-dm3f"
server_key: "0830197cbb0e241734b90fa71d8f0fd4"
server_key_iv: "47d046b3745714b990953956d30b0614"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:19.718142 27140 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.003s sys 0.004s
I20260619 18:55:19.744513 27160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:55:19.746826 27140 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.006s
I20260619 18:55:19.747200 27140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "97392ce8c68d4457acc58f3898677033"
format_stamp: "Formatted at 2026-06-19 18:55:19 on dist-test-slave-dm3f"
server_key: "0830197cbb0e241734b90fa71d8f0fd4"
server_key_iv: "47d046b3745714b990953956d30b0614"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:19.747704 27140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:55:19.782449 27140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:55:19.788373 27140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:55:19.788790 27140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:55:19.790030 27140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:55:19.792110 27140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:55:19.792239 27140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:55:19.792382 27140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:55:19.792472 27140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:55:19.892891 27140 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:38883
I20260619 18:55:19.892943 27272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:38883 every 8 connection(s)
I20260619 18:55:19.895048 27140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:55:19.903095 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 27140
I20260619 18:55:19.903342 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:55:19.904225 15724 external_mini_cluster.cc:1428] Setting key 221a335691240e3d1e93258d37a525fe
Jun 19 18:55:19 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895319, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
Jun 19 18:55:19 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895319, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:55:19.972932 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:19.921239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:47643 (local address 127.15.91.62:39087)
0619 18:55:19.944680 (+ 23441us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:19.944689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:19.948812 (+ 4123us) server_negotiation.cc:407] Connection header received
0619 18:55:19.948898 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:19.948903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:19.948999 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:19.949114 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:55:19.953819 (+ 4705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:19.955399 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:19.957777 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:19.958509 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:19.958671 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:19.964140 (+ 5469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:55:19.964149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:55:19.964154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:55:19.964224 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:55:19.968377 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:19.969640 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:19.969648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:19.969651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:19.969856 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:19.970519 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:19.970525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:19.970528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:19.970920 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:55:19.971076 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:19.972229 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0619 18:55:19.972533 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23249,"thread_start_us":23451,"threads_started":1}
I20260619 18:55:19.975718 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:19.920504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39087 (local address 127.15.91.1:47643)
0619 18:55:19.945504 (+ 25000us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:19.945577 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 18:55:19.948203 (+ 2626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:55:19.950644 (+ 2441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:55:19.950660 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:55:19.952376 (+ 1716us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:55:19.953464 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:19.953504 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:19.955726 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:19.955735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:19.957548 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:19.957570 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:19.958853 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:19.958864 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:19.959265 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:19.960233 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:55:19.960279 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:55:19.963674 (+ 3395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:55:19.968650 (+ 4976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:55:19.968662 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:55:19.968684 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:55:19.969431 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:55:19.970053 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:55:19.970058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:55:19.970061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:55:19.970377 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:55:19.971188 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:55:19.971198 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:55:19.971718 (+ 520us) client_negotiation.cc:769] Sending connection context
0619 18:55:19.973289 (+ 1571us) client_negotiation.cc:240] Negotiation successful
0619 18:55:19.973955 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24575,"thread_start_us":28626,"threads_started":1}
I20260619 18:55:19.978053 27273 heartbeater.cc:347] Connected to a master server at 127.15.91.62:39087
I20260619 18:55:19.978425 27273 heartbeater.cc:464] Registering TS with master...
I20260619 18:55:19.979415 27273 heartbeater.cc:511] Master 127.15.91.62:39087 requested a full tablet report, sending...
I20260619 18:55:19.982300 27060 ts_manager.cc:194] Registered new tserver with Master: 97392ce8c68d4457acc58f3898677033 (127.15.91.1:38883)
I20260619 18:55:19.984362 27060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:47643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:55:19.997709 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:51341/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39087
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:55:20 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895319, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
W20260619 18:55:20.551218 27279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:55:20.551651 27279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:55:20.551875 27279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:55:20.564337 27279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:55:20.564615 27279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:55:20.564829 27279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:55:20.581986 27279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.91.20:51341/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:55:20.583951 27279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:55:20.585929 27279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:55:20.601280 27296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:55:20.601372 27293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:55:20.601346 27294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:55:20.602540 27279 server_base.cc:1034] running on GCE node
I20260619 18:55:20.603379 27279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:55:20.604450 27279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:55:20.605713 27279 hybrid_clock.cc:648] HybridClock initialized: now 1781895320605624 us; error 67 us; skew 500 ppm
I20260619 18:55:20.606109 27279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:55:20 dist-test-slave-dm3f krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895320, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:55:20.610009 27279 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:55:20.611901 27279 webserver.cc:460] Webserver started at http://127.15.91.2:46313/ using document root <none> and password file <none>
I20260619 18:55:20.612686 27279 fs_manager.cc:359] Metadata directory not provided
I20260619 18:55:20.612818 27279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:55:20.613116 27279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:55:20 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895320, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:55:20.631191 27279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f"
format_stamp: "Formatted at 2026-06-19 18:55:20 on dist-test-slave-dm3f"
server_key: "c2d9a75ace2efa0baaf242567ef69476"
server_key_iv: "bbd2366cac560eacf1da7bfdffcd19a5"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:20.632238 27279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f"
format_stamp: "Formatted at 2026-06-19 18:55:20 on dist-test-slave-dm3f"
server_key: "c2d9a75ace2efa0baaf242567ef69476"
server_key_iv: "bbd2366cac560eacf1da7bfdffcd19a5"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:20.638931 27279 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260619 18:55:20.657716 27303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:55:20.659746 27279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:55:20.660039 27279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f"
format_stamp: "Formatted at 2026-06-19 18:55:20 on dist-test-slave-dm3f"
server_key: "c2d9a75ace2efa0baaf242567ef69476"
server_key_iv: "bbd2366cac560eacf1da7bfdffcd19a5"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:20.660375 27279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:55:20.680823 27279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:55:20.685845 27279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:55:20.686215 27279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:55:20.687458 27279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:55:20.689136 27279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:55:20.689229 27279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:55:20.689409 27279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:55:20.689464 27279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:55:20.729925 27279 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:40061
I20260619 18:55:20.729964 27415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:40061 every 8 connection(s)
I20260619 18:55:20.731493 27279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:55:20.735054 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 27279
I20260619 18:55:20.735260 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:55:20.735740 15724 external_mini_cluster.cc:1428] Setting key e8f38d70e404d02180d8687c54dcbe5c
Jun 19 18:55:20 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895320, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:55:20.755714 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:20.734471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:55085 (local address 127.15.91.62:39087)
0619 18:55:20.735166 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:20.735174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:20.736653 (+ 1479us) server_negotiation.cc:407] Connection header received
0619 18:55:20.738629 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:20.738637 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:20.738795 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:20.738917 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:55:20.740622 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:20.741898 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:20.743828 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:20.744266 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:20.744439 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:20.748727 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:55:20.748732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:55:20.748736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:55:20.748776 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:55:20.751849 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:20.752803 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:20.752808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:20.752810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:20.752949 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:20.753559 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:20.753563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:20.753565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:20.753884 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:55:20.754062 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:20.755006 (+ 944us) server_negotiation.cc:299] Negotiation successful
0619 18:55:20.755303 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":394,"threads_started":1}
I20260619 18:55:20.756814 27419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:20.735296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39087 (local address 127.15.91.2:55085)
0619 18:55:20.736378 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:20.736434 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:55:20.738289 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:55:20.739201 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:55:20.739212 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:55:20.739709 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:55:20.740381 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:20.740404 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:20.742095 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:20.742099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:20.743597 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:20.743616 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:20.744562 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:20.744570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:20.744990 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:20.745923 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:55:20.745976 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:55:20.748394 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:55:20.752071 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:55:20.752083 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:55:20.752107 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:55:20.752629 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:55:20.753082 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:55:20.753088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:55:20.753091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:55:20.753410 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:55:20.754035 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:55:20.754044 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:55:20.754550 (+ 506us) client_negotiation.cc:769] Sending connection context
0619 18:55:20.754881 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 18:55:20.755498 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":442,"threads_started":1}
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
I20260619 18:55:20.758692 27416 heartbeater.cc:347] Connected to a master server at 127.15.91.62:39087
I20260619 18:55:20.759110 27416 heartbeater.cc:464] Registering TS with master...
I20260619 18:55:20.760010 27416 heartbeater.cc:511] Master 127.15.91.62:39087 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
I20260619 18:55:20.761981 27060 ts_manager.cc:194] Registered new tserver with Master: 9f0dbe0fe62b420995e1c4d9c798fa9f (127.15.91.2:40061)
I20260619 18:55:20.762956 27060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:55085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:55:20.790274 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:51341/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39087
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:55:20.988858 27273 heartbeater.cc:503] Master 127.15.91.62:39087 was elected leader, sending a full tablet report...
W20260619 18:55:21.164834 27422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:55:21.165153 27422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:55:21.165333 27422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:55:21.173444 27422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:55:21.173658 27422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:55:21.173810 27422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:55:21.185469 27422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.91.20:51341/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:55:21.187034 27422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:55:21.189545 27422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:55:21.207414 27427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:55:21.207437 27430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:55:21.207368 27428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:55:21.208890 27422 server_base.cc:1034] running on GCE node
I20260619 18:55:21.209600 27422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:55:21.210865 27422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:55:21.212178 27422 hybrid_clock.cc:648] HybridClock initialized: now 1781895321212120 us; error 62 us; skew 500 ppm
I20260619 18:55:21.213012 27422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:55:21 dist-test-slave-dm3f krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:55:21.218189 27422 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:55:21.220311 27422 webserver.cc:460] Webserver started at http://127.15.91.3:46535/ using document root <none> and password file <none>
I20260619 18:55:21.221199 27422 fs_manager.cc:359] Metadata directory not provided
I20260619 18:55:21.221391 27422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:55:21.221755 27422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:55:21 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:55:21.237828 27422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "76cf3cd924834b919400ec457d3f7f1c"
format_stamp: "Formatted at 2026-06-19 18:55:21 on dist-test-slave-dm3f"
server_key: "2b25c2a022a5f47b3f2ac2a51337207f"
server_key_iv: "9c83fe77c01e8d6c8abed09da00f70fb"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:21.239027 27422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "76cf3cd924834b919400ec457d3f7f1c"
format_stamp: "Formatted at 2026-06-19 18:55:21 on dist-test-slave-dm3f"
server_key: "2b25c2a022a5f47b3f2ac2a51337207f"
server_key_iv: "9c83fe77c01e8d6c8abed09da00f70fb"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:21.245319 27422 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260619 18:55:21.261521 27437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:55:21.263419 27422 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:55:21.263612 27422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "76cf3cd924834b919400ec457d3f7f1c"
format_stamp: "Formatted at 2026-06-19 18:55:21 on dist-test-slave-dm3f"
server_key: "2b25c2a022a5f47b3f2ac2a51337207f"
server_key_iv: "9c83fe77c01e8d6c8abed09da00f70fb"
server_key_version: "kuduclusterkey@0"
I20260619 18:55:21.263896 27422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:55:21.291218 27422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:55:21.295497 27422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:55:21.295780 27422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:55:21.296906 27422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:55:21.298477 27422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:55:21.298568 27422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:55:21.298660 27422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:55:21.298705 27422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:55:21.336602 27422 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:39515
I20260619 18:55:21.336664 27549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:39515 every 8 connection(s)
I20260619 18:55:21.338070 27422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:55:21.347376 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 27422
I20260619 18:55:21.347561 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:55:21.347981 15724 external_mini_cluster.cc:1428] Setting key 010fe88a088fde511500e88f391d0a55
Jun 19 18:55:21 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:55:21.360718 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.341105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:59155 (local address 127.15.91.62:39087)
0619 18:55:21.341707 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:21.341711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:21.342642 (+ 931us) server_negotiation.cc:407] Connection header received
0619 18:55:21.344674 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:21.344680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:21.344788 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:21.344919 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:55:21.346594 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.347508 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.349561 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.350033 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.350188 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.353961 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:55:21.353967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:55:21.353971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:55:21.354009 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:55:21.356995 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:21.357967 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:21.357973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:21.357977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:21.358138 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:21.358666 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:21.358671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:21.358675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:21.359064 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:55:21.359242 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:21.360046 (+ 804us) server_negotiation.cc:299] Negotiation successful
0619 18:55:21.360330 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":330,"threads_started":1}
I20260619 18:55:21.361786 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.341595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39087 (local address 127.15.91.3:59155)
0619 18:55:21.342450 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.342503 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:55:21.344322 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:55:21.345212 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:55:21.345222 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:55:21.345707 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:55:21.346369 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.346389 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.347707 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.347712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.349387 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.349413 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.350326 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.350335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.350595 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.351426 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:55:21.351463 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:55:21.353646 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:55:21.357222 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:55:21.357234 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:55:21.357265 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:55:21.357793 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:55:21.358274 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:55:21.358280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:55:21.358284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:55:21.358532 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:55:21.359215 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:55:21.359224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:55:21.359634 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 18:55:21.359942 (+ 308us) client_negotiation.cc:240] Negotiation successful
0619 18:55:21.360540 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":340,"threads_started":1}
I20260619 18:55:21.363844 27550 heartbeater.cc:347] Connected to a master server at 127.15.91.62:39087
I20260619 18:55:21.364300 27550 heartbeater.cc:464] Registering TS with master...
I20260619 18:55:21.365202 27550 heartbeater.cc:511] Master 127.15.91.62:39087 requested a full tablet report, sending...
I20260619 18:55:21.367050 27059 ts_manager.cc:194] Registered new tserver with Master: 76cf3cd924834b919400ec457d3f7f1c (127.15.91.3:39515)
I20260619 18:55:21.368101 27059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:59155
I20260619 18:55:21.377820 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:55:21.400031 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.383237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.15.91.62:39087)
0619 18:55:21.383504 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:21.383514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:21.384024 (+ 510us) server_negotiation.cc:407] Connection header received
0619 18:55:21.384241 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:21.384248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:21.384381 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:21.384527 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:55:21.387222 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.388424 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.390315 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.390943 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.391108 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.393912 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:55:21.393919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:55:21.393923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:55:21.393968 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:55:21.396749 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:21.397716 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:21.397724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:21.397727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:21.397891 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:21.398351 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:21.398358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:21.398362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:21.398683 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:55:21.399035 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:21.399412 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 18:55:21.399687 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:55:21.404961 27059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:55:21.412523 27059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:55:21.444815 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.430116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.15.91.3:39515)
0619 18:55:21.431353 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:21.431363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:21.432792 (+ 1429us) server_negotiation.cc:407] Connection header received
0619 18:55:21.433036 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:21.433043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:21.433308 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:21.433540 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:55:21.436383 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.438428 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.441323 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.442908 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.443065 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.443351 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:21.444158 (+ 807us) server_negotiation.cc:299] Negotiation successful
0619 18:55:21.444378 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":890,"threads_started":1}
I20260619 18:55:21.445477 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.433651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.15.91.2:40061)
0619 18:55:21.434786 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:21.434800 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:21.435455 (+ 655us) server_negotiation.cc:407] Connection header received
0619 18:55:21.435824 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:21.435834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:21.436176 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:21.436422 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:55:21.437044 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.438604 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.442313 (+ 3709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.443804 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.443960 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.444172 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:21.444911 (+ 739us) server_negotiation.cc:299] Negotiation successful
0619 18:55:21.445126 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":1620,"threads_started":1}
I20260619 18:55:21.445523 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.430822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.15.91.1:38883)
0619 18:55:21.431854 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:21.431864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:21.434392 (+ 2528us) server_negotiation.cc:407] Connection header received
0619 18:55:21.434793 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:21.434803 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:21.435129 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:21.435382 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:55:21.436542 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.438805 (+ 2263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.442007 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.443706 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.443857 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.444129 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:21.444951 (+ 822us) server_negotiation.cc:299] Negotiation successful
0619 18:55:21.445150 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":741,"threads_started":1}
I20260619 18:55:21.445829 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.429985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:39515 (local address 127.0.0.1:53074)
0619 18:55:21.432417 (+ 2432us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.432502 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0619 18:55:21.432750 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:55:21.435583 (+ 2833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:55:21.435590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:55:21.435637 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:55:21.436098 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.436112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.438648 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.438654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.441084 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.441103 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.443535 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.443541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.443817 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.443898 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 18:55:21.445072 (+ 1174us) client_negotiation.cc:240] Negotiation successful
0619 18:55:21.445291 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2149,"mutex_wait_us":130,"spinlock_wait_cycles":3328,"thread_start_us":1977,"threads_started":1}
I20260619 18:55:21.446213 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.433374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:40061 (local address 127.0.0.1:49476)
0619 18:55:21.435338 (+ 1964us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.435360 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:55:21.435570 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:55:21.436426 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:55:21.436433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:55:21.436464 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:55:21.436821 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.436839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.439677 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.439685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.442117 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.442127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.444369 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.444375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.444698 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.444727 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:55:21.444859 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 18:55:21.445412 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1803,"mutex_wait_us":160,"thread_start_us":1634,"threads_started":1}
I20260619 18:55:21.446400 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.429999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:38883 (local address 127.0.0.1:57272)
0619 18:55:21.434256 (+ 4257us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.434277 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:55:21.434522 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:55:21.435590 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:55:21.435595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:55:21.435650 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:55:21.436309 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.436327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.438998 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.439007 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.441779 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.441797 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.444332 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.444339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.444704 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.444742 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 18:55:21.445746 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0619 18:55:21.445922 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4109,"mutex_wait_us":228,"spinlock_wait_cycles":10624,"thread_start_us":3917,"threads_started":1}
I20260619 18:55:21.450593 27485 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb2e8e0c974414ab58badd0bf579392 (DEFAULT_TABLE table=test-table [id=48bf6c58b2274d239c49c92575bde08f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:55:21.450588 27351 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb2e8e0c974414ab58badd0bf579392 (DEFAULT_TABLE table=test-table [id=48bf6c58b2274d239c49c92575bde08f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:55:21.450588 27208 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb2e8e0c974414ab58badd0bf579392 (DEFAULT_TABLE table=test-table [id=48bf6c58b2274d239c49c92575bde08f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:55:21.452312 27208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb2e8e0c974414ab58badd0bf579392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:55:21.452332 27351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb2e8e0c974414ab58badd0bf579392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:55:21.452567 27485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb2e8e0c974414ab58badd0bf579392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:55:21.466833 27569 tablet_bootstrap.cc:492] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: Bootstrap starting.
I20260619 18:55:21.470191 27570 tablet_bootstrap.cc:492] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f: Bootstrap starting.
I20260619 18:55:21.470948 27571 tablet_bootstrap.cc:492] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033: Bootstrap starting.
I20260619 18:55:21.471525 27569 tablet_bootstrap.cc:654] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: Neither blocks nor log segments found. Creating new log.
I20260619 18:55:21.473510 27569 log.cc:824] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:55:21.475761 27571 tablet_bootstrap.cc:654] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033: Neither blocks nor log segments found. Creating new log.
I20260619 18:55:21.477355 27571 log.cc:824] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033: Log is configured to *not* fsync() on all Append() calls
I20260619 18:55:21.478660 27570 tablet_bootstrap.cc:654] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f: Neither blocks nor log segments found. Creating new log.
I20260619 18:55:21.480300 27570 log.cc:824] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:55:21.482007 27571 tablet_bootstrap.cc:492] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033: No bootstrap required, opened a new log
I20260619 18:55:21.482498 27571 ts_tablet_manager.cc:1397] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260619 18:55:21.484248 27569 tablet_bootstrap.cc:492] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: No bootstrap required, opened a new log
I20260619 18:55:21.484666 27569 ts_tablet_manager.cc:1397] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.007s
I20260619 18:55:21.486189 27570 tablet_bootstrap.cc:492] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f: No bootstrap required, opened a new log
I20260619 18:55:21.486529 27570 ts_tablet_manager.cc:1397] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260619 18:55:21.494546 27570 raft_consensus.cc:348] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cf3cd924834b919400ec457d3f7f1c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39515 } } peers { permanent_uuid: "97392ce8c68d4457acc58f3898677033" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 38883 } } peers { permanent_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40061 } }
I20260619 18:55:21.495121 27570 raft_consensus.cc:374] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:55:21.495229 27570 raft_consensus.cc:729] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0dbe0fe62b420995e1c4d9c798fa9f, State: Initialized, Role: FOLLOWER
I20260619 18:55:21.495440 27571 raft_consensus.cc:348] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cf3cd924834b919400ec457d3f7f1c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39515 } } peers { permanent_uuid: "97392ce8c68d4457acc58f3898677033" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 38883 } } peers { permanent_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40061 } }
I20260619 18:55:21.496026 27570 consensus_queue.cc:260] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cf3cd924834b919400ec457d3f7f1c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39515 } } peers { permanent_uuid: "97392ce8c68d4457acc58f3898677033" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 38883 } } peers { permanent_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40061 } }
I20260619 18:55:21.496119 27571 raft_consensus.cc:374] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:55:21.496384 27571 raft_consensus.cc:729] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97392ce8c68d4457acc58f3898677033, State: Initialized, Role: FOLLOWER
I20260619 18:55:21.497202 27569 raft_consensus.cc:348] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cf3cd924834b919400ec457d3f7f1c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39515 } } peers { permanent_uuid: "97392ce8c68d4457acc58f3898677033" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 38883 } } peers { permanent_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40061 } }
I20260619 18:55:21.497471 27571 consensus_queue.cc:260] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cf3cd924834b919400ec457d3f7f1c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39515 } } peers { permanent_uuid: "97392ce8c68d4457acc58f3898677033" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 38883 } } peers { permanent_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40061 } }
I20260619 18:55:21.497751 27569 raft_consensus.cc:374] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:55:21.497852 27569 raft_consensus.cc:729] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76cf3cd924834b919400ec457d3f7f1c, State: Initialized, Role: FOLLOWER
I20260619 18:55:21.498693 27569 consensus_queue.cc:260] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cf3cd924834b919400ec457d3f7f1c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39515 } } peers { permanent_uuid: "97392ce8c68d4457acc58f3898677033" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 38883 } } peers { permanent_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40061 } }
I20260619 18:55:21.499676 27416 heartbeater.cc:503] Master 127.15.91.62:39087 was elected leader, sending a full tablet report...
I20260619 18:55:21.500428 27570 ts_tablet_manager.cc:1428] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260619 18:55:21.504149 27571 ts_tablet_manager.cc:1428] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033: Time spent starting tablet: real 0.021s user 0.018s sys 0.004s
I20260619 18:55:21.504954 27569 ts_tablet_manager.cc:1428] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: Time spent starting tablet: real 0.020s user 0.015s sys 0.005s
I20260619 18:55:21.505651 27550 heartbeater.cc:503] Master 127.15.91.62:39087 was elected leader, sending a full tablet report...
I20260619 18:55:21.514644 27576 raft_consensus.cc:482] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:55:21.515051 27576 raft_consensus.cc:504] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cf3cd924834b919400ec457d3f7f1c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39515 } } peers { permanent_uuid: "97392ce8c68d4457acc58f3898677033" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 38883 } } peers { permanent_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40061 } }
I20260619 18:55:21.516973 27576 leader_election.cc:288] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76cf3cd924834b919400ec457d3f7f1c (127.15.91.3:39515), 9f0dbe0fe62b420995e1c4d9c798fa9f (127.15.91.2:40061)
I20260619 18:55:21.525910 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.517536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:48463 (local address 127.15.91.3:39515)
0619 18:55:21.517734 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:21.517739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:21.518493 (+ 754us) server_negotiation.cc:407] Connection header received
0619 18:55:21.518802 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:21.518809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:21.518897 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:21.519025 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:55:21.519647 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.520391 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.523304 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.524637 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.524831 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.524956 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:21.525453 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 18:55:21.525602 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:55:21.525975 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.518902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:36067 (local address 127.15.91.2:40061)
0619 18:55:21.519214 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:21.519221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:21.519305 (+ 84us) server_negotiation.cc:407] Connection header received
0619 18:55:21.519523 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:21.519533 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:21.519607 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:21.519709 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:55:21.520271 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.520969 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.523601 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.524862 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.525014 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.525122 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:21.525516 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 18:55:21.525670 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260619 18:55:21.526036 27578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.517451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:39515 (local address 127.15.91.1:48463)
0619 18:55:21.518202 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.518222 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:55:21.518402 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:55:21.519077 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:55:21.519085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:55:21.519116 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:55:21.519482 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.519492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.520534 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.520541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.522979 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.523007 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.525062 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.525070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.525240 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.525271 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:55:21.525418 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 18:55:21.525590 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":421,"threads_started":1}
I20260619 18:55:21.526064 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.518431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:40061 (local address 127.15.91.1:36067)
0619 18:55:21.519188 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.519209 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:55:21.519364 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:55:21.519726 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:55:21.519730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:55:21.519757 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:55:21.520112 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.520121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.521200 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.521211 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.523388 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:55:21.523400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.525170 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.525182 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:55:21.525331 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.525356 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:55:21.525472 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 18:55:21.525681 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":397,"threads_started":1}
I20260619 18:55:21.527503 27371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb2e8e0c974414ab58badd0bf579392" candidate_uuid: "97392ce8c68d4457acc58f3898677033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" is_pre_election: true
I20260619 18:55:21.527513 27505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb2e8e0c974414ab58badd0bf579392" candidate_uuid: "97392ce8c68d4457acc58f3898677033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76cf3cd924834b919400ec457d3f7f1c" is_pre_election: true
I20260619 18:55:21.528054 27505 raft_consensus.cc:2454] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97392ce8c68d4457acc58f3898677033 in term 0.
I20260619 18:55:21.528055 27371 raft_consensus.cc:2454] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97392ce8c68d4457acc58f3898677033 in term 0.
I20260619 18:55:21.528860 27162 leader_election.cc:302] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76cf3cd924834b919400ec457d3f7f1c, 97392ce8c68d4457acc58f3898677033; no voters:
I20260619 18:55:21.529345 27576 raft_consensus.cc:2781] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:55:21.529517 27576 raft_consensus.cc:482] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:55:21.529631 27576 raft_consensus.cc:3037] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:55:21.532090 27576 raft_consensus.cc:504] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cf3cd924834b919400ec457d3f7f1c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39515 } } peers { permanent_uuid: "97392ce8c68d4457acc58f3898677033" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 38883 } } peers { permanent_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40061 } }
I20260619 18:55:21.532831 27576 leader_election.cc:288] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [CANDIDATE]: Term 1 election: Requested vote from peers 76cf3cd924834b919400ec457d3f7f1c (127.15.91.3:39515), 9f0dbe0fe62b420995e1c4d9c798fa9f (127.15.91.2:40061)
I20260619 18:55:21.533421 27505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb2e8e0c974414ab58badd0bf579392" candidate_uuid: "97392ce8c68d4457acc58f3898677033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76cf3cd924834b919400ec457d3f7f1c"
I20260619 18:55:21.533537 27371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb2e8e0c974414ab58badd0bf579392" candidate_uuid: "97392ce8c68d4457acc58f3898677033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f"
I20260619 18:55:21.533674 27505 raft_consensus.cc:3037] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:55:21.533792 27371 raft_consensus.cc:3037] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:55:21.536305 27505 raft_consensus.cc:2454] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97392ce8c68d4457acc58f3898677033 in term 1.
I20260619 18:55:21.536749 27371 raft_consensus.cc:2454] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97392ce8c68d4457acc58f3898677033 in term 1.
I20260619 18:55:21.536922 27162 leader_election.cc:302] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76cf3cd924834b919400ec457d3f7f1c, 97392ce8c68d4457acc58f3898677033; no voters:
I20260619 18:55:21.537349 27576 raft_consensus.cc:2781] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:55:21.537894 27576 raft_consensus.cc:686] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 1 LEADER]: Becoming Leader. State: Replica: 97392ce8c68d4457acc58f3898677033, State: Running, Role: LEADER
I20260619 18:55:21.538434 27576 consensus_queue.cc:237] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cf3cd924834b919400ec457d3f7f1c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39515 } } peers { permanent_uuid: "97392ce8c68d4457acc58f3898677033" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 38883 } } peers { permanent_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40061 } }
I20260619 18:55:21.543995 27059 catalog_manager.cc:5495] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97392ce8c68d4457acc58f3898677033 (127.15.91.1). New cstate: current_term: 1 leader_uuid: "97392ce8c68d4457acc58f3898677033" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cf3cd924834b919400ec457d3f7f1c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97392ce8c68d4457acc58f3898677033" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40061 } health_report { overall_health: UNKNOWN } } }
I20260619 18:55:21.574290 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.567922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.15.91.1:38883)
0619 18:55:21.568167 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:21.568173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:21.568193 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:55:21.568429 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:21.568435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:21.568531 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:21.568708 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:55:21.569155 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.570078 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.571782 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.572233 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.572389 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.573002 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:55:21.573570 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:55:21.573751 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:21.573900 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 18:55:21.573998 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:55:21.584981 27505 raft_consensus.cc:1261] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c [term 1 FOLLOWER]: Refusing update from remote peer 97392ce8c68d4457acc58f3898677033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:55:21.584981 27371 raft_consensus.cc:1261] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f [term 1 FOLLOWER]: Refusing update from remote peer 97392ce8c68d4457acc58f3898677033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:55:21.586236 27576 consensus_queue.cc:1035] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f0dbe0fe62b420995e1c4d9c798fa9f" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:55:21.586632 27580 consensus_queue.cc:1035] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76cf3cd924834b919400ec457d3f7f1c" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:55:21.589910 27551 tablet.cc:2369] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:55:21.601874 27583 mvcc.cc:205] Tried to move back new op lower bound from 7298643237198319616 to 7298643237025898496. Current Snapshot: MvccSnapshot[applied={T|T < 7298643237198319616}]
I20260619 18:55:21.617837 27059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:55:21.618171 27059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:55:21.622515 27059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d77e28316cc4f1dabbf0c982909a756: Sending DeleteTablet for 3 replicas of tablet 9fb2e8e0c974414ab58badd0bf579392
I20260619 18:55:21.623981 27208 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fb2e8e0c974414ab58badd0bf579392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:55:21 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:57272
I20260619 18:55:21.624023 27485 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fb2e8e0c974414ab58badd0bf579392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:55:21 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:53074
I20260619 18:55:21.624089 27351 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fb2e8e0c974414ab58badd0bf579392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:55:21 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:49476
I20260619 18:55:21.628616 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 27140
I20260619 18:55:21.637144 27594 tablet_replica.cc:330] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f: stopping tablet replica
I20260619 18:55:21.637876 27594 raft_consensus.cc:2229] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:55:21.638568 27594 raft_consensus.cc:2258] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:55:21.639575 27595 tablet_replica.cc:330] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: stopping tablet replica
I20260619 18:55:21.640085 27595 raft_consensus.cc:2229] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:55:21.640632 27595 raft_consensus.cc:2258] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:55:21.641654 27594 ts_tablet_manager.cc:1905] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:55:21.644908 27595 ts_tablet_manager.cc:1905] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:55:21 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:55:21.649837 27594 ts_tablet_manager.cc:1918] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:55:21.650081 27594 log.cc:1196] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/9fb2e8e0c974414ab58badd0bf579392
I20260619 18:55:21.651139 27593 tablet_replica.cc:330] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033: stopping tablet replica
I20260619 18:55:21.651708 27593 raft_consensus.cc:2229] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:55:21.652534 27593 raft_consensus.cc:2258] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:55:21.652619 27595 ts_tablet_manager.cc:1918] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:55:21.652791 27595 log.cc:1196] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/9fb2e8e0c974414ab58badd0bf579392
I20260619 18:55:21.653633 27595 ts_tablet_manager.cc:1939] T 9fb2e8e0c974414ab58badd0bf579392 P 76cf3cd924834b919400ec457d3f7f1c: Deleting consensus metadata
I20260619 18:55:21.655149 27593 ts_tablet_manager.cc:1905] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:55:21.655928 27045 catalog_manager.cc:4841] TS 76cf3cd924834b919400ec457d3f7f1c (127.15.91.3:39515): tablet 9fb2e8e0c974414ab58badd0bf579392 (table test-table [id=48bf6c58b2274d239c49c92575bde08f]) successfully deleted
I20260619 18:55:21.656561 27594 ts_tablet_manager.cc:1939] T 9fb2e8e0c974414ab58badd0bf579392 P 9f0dbe0fe62b420995e1c4d9c798fa9f: Deleting consensus metadata
I20260619 18:55:21.658802 27045 catalog_manager.cc:4841] TS 9f0dbe0fe62b420995e1c4d9c798fa9f (127.15.91.2:40061): tablet 9fb2e8e0c974414ab58badd0bf579392 (table test-table [id=48bf6c58b2274d239c49c92575bde08f]) successfully deleted
I20260619 18:55:21.661640 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.630783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.15.91.1:38883)
0619 18:55:21.631006 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:21.631013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:21.632678 (+ 1665us) server_negotiation.cc:407] Connection header received
0619 18:55:21.634068 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:21.634078 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:21.634207 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:21.634324 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:55:21.638791 (+ 4467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.640096 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.642694 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.643470 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.643619 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.651848 (+ 8229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:55:21.651871 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:55:21.651892 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:55:21.651966 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:55:21.656833 (+ 4867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:21.658088 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:21.658100 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:21.658105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:21.658323 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:21.658933 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:21.658939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:21.658944 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:21.659534 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:55:21.659893 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:21.660745 (+ 852us) server_negotiation.cc:299] Negotiation successful
0619 18:55:21.661050 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:55:21.663059 27268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:55:21.766750 27593 ts_tablet_manager.cc:1918] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:55:21.767063 27593 log.cc:1196] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/9fb2e8e0c974414ab58badd0bf579392
I20260619 18:55:21.767688 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 27140
I20260619 18:55:21.767853 27593 ts_tablet_manager.cc:1939] T 9fb2e8e0c974414ab58badd0bf579392 P 97392ce8c68d4457acc58f3898677033: Deleting consensus metadata
I20260619 18:55:21.788102 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 27279
W20260619 18:55:21.788110 27046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.91.1:38883 (error 108)
I20260619 18:55:21.790043 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.789614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:38883 (local address 127.0.0.1:57298)
0619 18:55:21.789789 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.789900 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 18:55:21.790292 27046 catalog_manager.cc:4584] TS 97392ce8c68d4457acc58f3898677033 (127.15.91.1:38883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fb2e8e0c974414ab58badd0bf579392: Network error: Client connection negotiation failed: client connection to 127.15.91.1:38883: connect: Connection refused (error 111)
Jun 19 18:55:21 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:55:21.805935 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.789049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.15.91.2:40061)
0619 18:55:21.789216 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:21.789222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:21.789273 (+ 51us) server_negotiation.cc:407] Connection header received
0619 18:55:21.789524 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:21.789534 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:21.789620 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:21.789748 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:55:21.791433 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.792214 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.793683 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.794103 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.794271 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.799271 (+ 5000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:55:21.799289 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:55:21.799307 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:55:21.799351 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:55:21.802461 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:21.803472 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:21.803481 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:21.803484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:21.803647 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:21.804131 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:21.804136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:21.804138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:21.804553 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:55:21.804905 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:21.805284 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 18:55:21.805544 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:55:21.806814 27411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:55:21.843917 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.843435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:38883 (local address 127.0.0.1:57312)
0619 18:55:21.843667 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.843751 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 18:55:21.845018 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.844641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:38883 (local address 127.0.0.1:57326)
0619 18:55:21.844800 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.844895 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 18:55:21.901398 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 27279
I20260619 18:55:21.905938 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.905403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:38883 (local address 127.0.0.1:57332)
0619 18:55:21.905679 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.905772 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 18:55:21.907137 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.906705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:38883 (local address 127.0.0.1:57342)
0619 18:55:21.906929 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.907004 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 18:55:21.919243 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 27422
Jun 19 18:55:21 dist-test-slave-dm3f krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:55:21.937084 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.920055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.15.91.3:39515)
0619 18:55:21.920230 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:55:21.920235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:55:21.920255 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:55:21.920402 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:55:21.920406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:55:21.920472 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:55:21.920565 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:55:21.922050 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.923107 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.924578 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:55:21.925007 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:55:21.925131 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:55:21.930966 (+ 5835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:55:21.930984 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:55:21.931003 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:55:21.931055 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:55:21.933946 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:21.934905 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:21.934913 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:21.934915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:21.935055 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:55:21.935519 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:55:21.935522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:55:21.935524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:55:21.935940 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:55:21.936155 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 18:55:21.936550 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 18:55:21.936797 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:55:21.937804 27545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:55:21.987844 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.987348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:38883 (local address 127.0.0.1:57354)
0619 18:55:21.987591 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.987688 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 18:55:21.989185 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:21.988725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:38883 (local address 127.0.0.1:57366)
0619 18:55:21.988941 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:21.989053 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260619 18:55:22.031077 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 27422
I20260619 18:55:22.048333 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 27028
I20260619 18:55:22.049723 27090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:55:22.167871 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:55:22.166795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:38883 (local address 127.0.0.1:57380)
0619 18:55:22.167335 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 18:55:22.167475 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361}
I20260619 18:55:22.167974 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 27028
I20260619 18:55:22.191087 15724 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 18:55:22.527941 15724 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 18:55:22.528219 15724 mini_ranger.cc:66] Stopping Ranger...
I20260619 18:55:22.875136 15724 mini_ranger.cc:68] Stopped Ranger
2026-06-19 18:55:22.875 UTC [26614] LOG: received smart shutdown request
2026-06-19 18:55:22.879 UTC [26614] LOG: background worker "logical replication launcher" (PID 26623) exited with exit code 1
2026-06-19 18:55:22.879 UTC [26618] LOG: shutting down
2026-06-19 18:55:23.049 UTC [26614] LOG: database system is shut down
2026-06-19T18:55:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55440 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:55:23 dist-test-slave-dm3f krb5kdc[27610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:55:23 dist-test-slave-dm3f krb5kdc[27610](info): set up 2 sockets
Jun 19 18:55:23 dist-test-slave-dm3f krb5kdc[27610](info): commencing operation
krb5kdc: starting...
W20260619 18:55:24.319288 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:55:24 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:55:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:55:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.91.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.91.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/rangeradmin_127.15.91.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/rangeradmin_127.15.91.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.91.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.91.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/rangerlookup_127.15.91.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/rangerlookup_127.15.91.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.91.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.91.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/HTTP_127.15.91.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/HTTP_127.15.91.20@KRBTEST.COM.keytab.
I20260619 18:55:24.679427 15724 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskualRwQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/postgres -l logfile start
2026-06-19 18:55:27.490 UTC [27640] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 18:55:27.490 UTC [27640] LOG: listening on IPv4 address "127.15.91.20", port 49857
2026-06-19 18:55:27.495 UTC [27640] LOG: listening on Unix socket "/tmp/.s.PGSQL.49857"
2026-06-19 18:55:27.518 UTC [27643] LOG: database system was shut down at 2026-06-19 18:55:25 UTC
2026-06-19 18:55:27.523 UTC [27640] LOG: database system is ready to accept connections
I20260619 18:55:28.490990 15724 mini_postgres.cc:96] Postgres bound to 49857
2026-06-19 18:55:28.508 UTC [27651] FATAL: database "slave" does not exist
127.15.91.20:49857 - accepting connections
I20260619 18:55:28.509675 15724 mini_ranger.cc:161] Starting Ranger...
I20260619 18:55:28.538144 15724 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 18:55:29.139243 15724 mini_ranger.cc:87] Created ranger Postgres database
I20260619 18:55:29.139424 15724 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-admin
2026-06-19 18:55:29,447 [I] DB FLAVOR :POSTGRES
2026-06-19 18:55:29,447 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:55:29,447 [I] Checking connection..
2026-06-19 18:55:29,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 18:55:29,726 [I] Checking connection passed.
2026-06-19 18:55:29,726 [I] --------- Verifying version history table ---------
2026-06-19 18:55:29,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:55:29,985 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:55:29,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:55:30,277 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:55:30,277 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 18:55:30,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 18:55:30.530 UTC [27715] WARNING: there is no transaction in progress
2026-06-19 18:55:30,540 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 18:55:30,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:55:30,821 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 18:55:30,822 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:55:30,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 18:55:31,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 18:55:31,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dm3f.c.gcp-upstream.internal','N') ;"
2026-06-19 18:55:31,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:55:31,937 [I] Table x_portal_user does not exist in database ranger
2026-06-19 18:55:31,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:55:32,207 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 18:55:32,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:55:32,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:55:32,734 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 18:55:32,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 18:55:33.796 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:33.815 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:33.833 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:33.851 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:33.868 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:33.888 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:33.951 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:33.959 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:33.976 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:33.988 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:33.999 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:34.015 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:34.026 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:34.034 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:34.042 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:34.052 UTC [27823] WARNING: there is no transaction in progress
2026-06-19 18:55:34,821 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 18:55:34,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 18:55:35,075 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 18:55:35,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:55:35,344 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 18:55:35,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:55:35,615 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 18:55:35,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:55:35,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:55:36,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dm3f.c.gcp-upstream.internal';"
2026-06-19 18:55:36,411 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 18:55:36,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 18:55:36,669 [I] DB_PATCHES have already been applied
I20260619 18:55:36.675832 15724 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-admin:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskualRwQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 18:55:36.675936 15724 mini_ranger.cc:193] Using host: 127.15.91.20
I20260619 18:55:36.691538 15724 mini_ranger.cc:227] Ranger admin URL: http://127.15.91.20:39603
Jun 19, 2026 6:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 18:55:37.852897 15724 mini_ranger.cc:160] Time spent starting Ranger: real 9.343s user 0.002s sys 0.041s
Jun 19, 2026 6:56:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:56:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:56:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:56:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:56:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 18:56:03.481696 15724 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.91.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.91.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/rangerkms_127.15.91.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.91.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/rangerkms_127.15.91.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.91.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.91.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.91.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/HTTP_127.15.91.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.91.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/HTTP_127.15.91.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 18:56:03 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895363, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 18:56:03.653903 15724 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 18:56:03.677407 15724 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 18:56:04.559286 15724 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 18:56:04.559417 15724 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-kms
I20260619 18:56:04.559473 15724 mini_ranger_kms.cc:227] Using postgres at 127.15.91.20:49857
2026-06-19 18:56:04,946 [I] DB FLAVOR :POSTGRES
2026-06-19 18:56:04,946 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:56:04,946 [I] Checking connection
2026-06-19 18:56:04,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 18:56:05,223 [I] connection success
2026-06-19 18:56:05,223 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 18:56:05,223 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 18:56:05,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 18:56:05,509 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 18:56:05,510 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:56:05,510 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 18:56:05,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.91.20:49857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 18:56:05,809 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 18:56:06.381191 15724 mini_ranger_kms.cc:310] Created kms service
I20260619 18:56:06.619817 15724 mini_ranger_kms.cc:326] Created kudu user
I20260619 18:56:06.673940 15724 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 18:56:06 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895363, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:56:07.172503 15724 mini_ranger_kms.cc:384] Added ranger policy
I20260619 18:56:07.172852 15724 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-kms:/tmp/dist-test-taskualRwQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskualRwQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 18:56:07.172945 15724 mini_ranger_kms.cc:240] Using host: 127.15.91.20
I20260619 18:56:07.191636 15724 mini_ranger_kms.cc:276] Ranger KMS PID: 28004
I20260619 18:56:07.191789 15724 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.91.20:54383
Jun 19, 2026 6:56:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:56:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:56:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:56:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:56:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:56:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 18:56:09.501569 15724 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.848s user 0.082s sys 0.321s
I20260619 18:56:09.501847 15724 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 18:56:09.502033 15724 mini_ranger_kms.cc:401] 127.15.91.20:54383/kms/v1/keys
Jun 19 18:56:09 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781895369, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.91.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 18:56:10 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895369, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.91.20@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
Jun 19 18:56:10 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895369, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.91.20@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
Jun 19 18:56:13 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:56:13.219067 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:42901
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:42901
--ranger_config_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:54383/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:13.596657 28054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:13.597007 28054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:13.597154 28054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:13.605257 28054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:56:13.605400 28054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:13.605448 28054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:13.605569 28054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:56:13.605633 28054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:56:13.616869 28054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.91.20:54383/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:42901
--ranger_config_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:42901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:13.618348 28054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:13.619904 28054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:13.633410 28059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:13.633334 28060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:13.633359 28062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:13.634130 28054 server_base.cc:1034] running on GCE node
I20260619 18:56:13.635205 28054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:13.637193 28054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:13.638475 28054 hybrid_clock.cc:648] HybridClock initialized: now 1781895373638420 us; error 52 us; skew 500 ppm
I20260619 18:56:13.639025 28054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:13 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895373, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:13.642809 28054 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:56:13.644721 28054 webserver.cc:460] Webserver started at http://127.15.91.62:33237/ using document root <none> and password file <none>
I20260619 18:56:13.645551 28054 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:13.645720 28054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:13.646088 28054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:56:13 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895373, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:56:13.714330 28054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "85a857adf9bb436abbc2b14d9cd1a8d3"
format_stamp: "Formatted at 2026-06-19 18:56:13 on dist-test-slave-dm3f"
server_key: "e1d837c46d89019296c3fa12b80e5694"
server_key_iv: "456b204b861a1f3a2129d571ad3ca240"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:13.715869 28054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "85a857adf9bb436abbc2b14d9cd1a8d3"
format_stamp: "Formatted at 2026-06-19 18:56:13 on dist-test-slave-dm3f"
server_key: "e1d837c46d89019296c3fa12b80e5694"
server_key_iv: "456b204b861a1f3a2129d571ad3ca240"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:13.724792 28054 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260619 18:56:13.745797 28069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:13.748083 28054 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260619 18:56:13.748353 28054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "85a857adf9bb436abbc2b14d9cd1a8d3"
format_stamp: "Formatted at 2026-06-19 18:56:13 on dist-test-slave-dm3f"
server_key: "e1d837c46d89019296c3fa12b80e5694"
server_key_iv: "456b204b861a1f3a2129d571ad3ca240"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:13.748638 28054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:13.771564 28054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:13.776676 28054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:13.777110 28054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:13.799022 28054 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:42901
I20260619 18:56:13.799054 28120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:42901 every 8 connection(s)
I20260619 18:56:13.800675 28054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:56:13.806205 28121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:13.807430 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 28054
I20260619 18:56:13.807679 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:56:13.808547 15724 external_mini_cluster.cc:1428] Setting key cbf21dee47a32bb8bce9d03892247cbe
I20260619 18:56:13.822271 28121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3: Bootstrap starting.
I20260619 18:56:13.826107 28121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:13.827711 28121 log.cc:824] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:56:13 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:13.831487 28121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3: No bootstrap required, opened a new log
I20260619 18:56:13.840641 28121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } }
I20260619 18:56:13.841050 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:13.817090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.15.91.62:42901)
0619 18:56:13.818216 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:13.818231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:13.818284 (+ 53us) server_negotiation.cc:407] Connection header received
0619 18:56:13.820565 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:13.820599 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:13.821148 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:13.821631 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:13.823480 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:13.824956 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:13.826592 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:13.827528 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:13.827773 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:13.832539 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:13.832548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:13.832567 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:13.832612 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:13.836145 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:13.837103 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:13.837113 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:13.837122 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:13.837361 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:13.837833 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:13.837837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:13.837839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:13.838504 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:13.838884 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:13.839434 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 18:56:13.839930 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":559,"threads_started":1}
I20260619 18:56:13.841113 28121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:13.841236 28121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a857adf9bb436abbc2b14d9cd1a8d3, State: Initialized, Role: FOLLOWER
I20260619 18:56:13.842062 28121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } }
I20260619 18:56:13.842355 28121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:56:13.842492 28121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:56:13.842674 28121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:13.845510 28121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } }
I20260619 18:56:13.846165 28121 leader_election.cc:302] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85a857adf9bb436abbc2b14d9cd1a8d3; no voters:
I20260619 18:56:13.846896 28121 leader_election.cc:288] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:56:13.847072 28126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:56:13.848419 28126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 1 LEADER]: Becoming Leader. State: Replica: 85a857adf9bb436abbc2b14d9cd1a8d3, State: Running, Role: LEADER
I20260619 18:56:13.849061 28126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } }
I20260619 18:56:13.849421 28121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:56:13.853330 28127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } } }
I20260619 18:56:13.853745 28127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:13.855113 28128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a857adf9bb436abbc2b14d9cd1a8d3. Latest consensus state: current_term: 1 leader_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } } }
I20260619 18:56:13.855564 28128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:13.855661 28134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:56:13.862071 28134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:56:13.862985 28121 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:56:13.888813 28134 catalog_manager.cc:1347] Generated new cluster ID: ea39f97941194e409ce4ebec2d6dff95
I20260619 18:56:13.889001 28134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:56:13.906922 28134 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:56:13.908500 28134 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:56:13.917040 28134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3: Generated new TSK 0
I20260619 18:56:13.918202 28134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:56:14.807482 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:54383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42901
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:56:15 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781895375, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 18:56:15.233489 28166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:15.233862 28166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:15.234027 28166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:15.242446 28166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:15.242654 28166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:15.242874 28166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:56:15.256141 28166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.91.20:54383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:15.257892 28166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:15.259634 28166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:15.279893 28176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:15.281795 28177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:15.287032 28179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:15.287921 28166 server_base.cc:1034] running on GCE node
I20260619 18:56:15.288776 28166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:15.290073 28166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:15.291334 28166 hybrid_clock.cc:648] HybridClock initialized: now 1781895375291261 us; error 73 us; skew 500 ppm
I20260619 18:56:15.291780 28166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:15 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895375, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:15.295475 28166 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:56:15.304673 28166 webserver.cc:460] Webserver started at http://127.15.91.1:38081/ using document root <none> and password file <none>
I20260619 18:56:15.305627 28166 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:15.305809 28166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:15.306243 28166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:56:15 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895375, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:56:15.326395 28166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "d4467accee9549949396a3ab71bb9cea"
format_stamp: "Formatted at 2026-06-19 18:56:15 on dist-test-slave-dm3f"
server_key: "daec6e2a650ba3d126309c3ce622509d"
server_key_iv: "707a48d366154c67384f57ced8dde6b2"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:15.328104 28166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "d4467accee9549949396a3ab71bb9cea"
format_stamp: "Formatted at 2026-06-19 18:56:15 on dist-test-slave-dm3f"
server_key: "daec6e2a650ba3d126309c3ce622509d"
server_key_iv: "707a48d366154c67384f57ced8dde6b2"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:15.346692 28166 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.012s sys 0.004s
I20260619 18:56:15.391527 28186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:15.395113 28166 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.015s sys 0.000s
I20260619 18:56:15.395612 28166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "d4467accee9549949396a3ab71bb9cea"
format_stamp: "Formatted at 2026-06-19 18:56:15 on dist-test-slave-dm3f"
server_key: "daec6e2a650ba3d126309c3ce622509d"
server_key_iv: "707a48d366154c67384f57ced8dde6b2"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:15.396301 28166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:15.443544 28166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:15.449208 28166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:15.449613 28166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:15.450987 28166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:15.453121 28166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:15.453248 28166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:15.453375 28166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:15.453438 28166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:15.553654 28166 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:43253
I20260619 18:56:15.553802 28298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:43253 every 8 connection(s)
I20260619 18:56:15.555583 28166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:56:15.562546 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 28166
I20260619 18:56:15.562973 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:56:15.564047 15724 external_mini_cluster.cc:1428] Setting key f0c644004f2189fb0c1ab616cc087ab7
Jun 19 18:56:15 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895375, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:56:15.606051 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:15.558889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:41183 (local address 127.15.91.62:42901)
0619 18:56:15.571331 (+ 12442us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:15.571340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:15.577509 (+ 6169us) server_negotiation.cc:407] Connection header received
0619 18:56:15.580670 (+ 3161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:15.580678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:15.580807 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:15.580902 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:15.583907 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:15.585157 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:15.588179 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:15.588887 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:15.589049 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:15.594491 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:15.594500 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:15.594505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:15.594557 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:15.598574 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:15.600119 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:15.600126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:15.600130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:15.600342 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:15.601046 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:15.601052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:15.601056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:15.601452 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:15.601614 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:15.605339 (+ 3725us) server_negotiation.cc:299] Negotiation successful
0619 18:56:15.605640 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12275,"thread_start_us":12130,"threads_started":1}
I20260619 18:56:15.607609 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:15.559836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42901 (local address 127.15.91.1:41183)
0619 18:56:15.577248 (+ 17412us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:15.577332 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:15.580203 (+ 2871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:15.581610 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:15.581625 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:15.582514 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:15.583570 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:15.583606 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:15.585460 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:15.585466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:15.587883 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:15.587911 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:15.589530 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:15.589541 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:15.589905 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:15.590942 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:15.590992 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:15.594113 (+ 3121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:15.598852 (+ 4739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:15.598865 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:15.598888 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:15.599651 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:15.600541 (+ 890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:15.600548 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:15.600552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:15.600878 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:15.603478 (+ 2600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:15.603488 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:15.604646 (+ 1158us) client_negotiation.cc:769] Sending connection context
0619 18:56:15.605101 (+ 455us) client_negotiation.cc:240] Negotiation successful
0619 18:56:15.605760 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16983,"thread_start_us":16663,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:56:15.609745 28299 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42901
I20260619 18:56:15.610142 28299 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:15.611183 28299 heartbeater.cc:511] Master 127.15.91.62:42901 requested a full tablet report, sending...
I20260619 18:56:15.619061 28086 ts_manager.cc:194] Registered new tserver with Master: d4467accee9549949396a3ab71bb9cea (127.15.91.1:43253)
I20260619 18:56:15.621382 28086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:41183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:56:15.646173 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:54383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42901
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:56:15 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895375, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
Jun 19 18:56:16 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895375, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
W20260619 18:56:16.134948 28305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:16.135432 28305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:16.135624 28305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:16.148029 28305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:16.148231 28305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:16.148435 28305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:56:16.165750 28305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.91.20:54383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:16.167737 28305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:16.169710 28305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:16.204492 28313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:16.206885 28312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:16.207216 28315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:16.208596 28305 server_base.cc:1034] running on GCE node
I20260619 18:56:16.209523 28305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:16.220001 28305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:16.221381 28305 hybrid_clock.cc:648] HybridClock initialized: now 1781895376221289 us; error 81 us; skew 500 ppm
I20260619 18:56:16.221870 28305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:16 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895376, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:16.230772 28305 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:56:16.235085 28305 webserver.cc:460] Webserver started at http://127.15.91.2:40849/ using document root <none> and password file <none>
I20260619 18:56:16.236382 28305 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:16.236593 28305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:16.237025 28305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:56:16 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895376, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:56:16.271168 28305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "d754e40fd4b245b7bd44aa79aaab41fb"
format_stamp: "Formatted at 2026-06-19 18:56:16 on dist-test-slave-dm3f"
server_key: "bdace31f8d52382120aa26cc59a3074f"
server_key_iv: "0cfb309d069bb8a70ef0701fb601b767"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:16.273651 28305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "d754e40fd4b245b7bd44aa79aaab41fb"
format_stamp: "Formatted at 2026-06-19 18:56:16 on dist-test-slave-dm3f"
server_key: "bdace31f8d52382120aa26cc59a3074f"
server_key_iv: "0cfb309d069bb8a70ef0701fb601b767"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:16.290330 28305 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.012s sys 0.004s
I20260619 18:56:16.310132 28322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:16.312201 28305 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:56:16.312463 28305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "d754e40fd4b245b7bd44aa79aaab41fb"
format_stamp: "Formatted at 2026-06-19 18:56:16 on dist-test-slave-dm3f"
server_key: "bdace31f8d52382120aa26cc59a3074f"
server_key_iv: "0cfb309d069bb8a70ef0701fb601b767"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:16.312740 28305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:16.362403 28305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:16.367238 28305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:16.367586 28305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:16.368738 28305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:16.370350 28305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:16.370474 28305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:16.370642 28305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:16.370714 28305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:16.417176 28305 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:34921
I20260619 18:56:16.417244 28441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:34921 every 8 connection(s)
I20260619 18:56:16.419045 28305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:56:16.423213 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 28305
I20260619 18:56:16.423403 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:56:16.423990 15724 external_mini_cluster.cc:1428] Setting key 9786c935a778120b0a800ce673892d65
Jun 19 18:56:16 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895376, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
I20260619 18:56:16.462288 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:16.422963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:41867 (local address 127.15.91.62:42901)
0619 18:56:16.438330 (+ 15367us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:16.438341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:16.442663 (+ 4322us) server_negotiation.cc:407] Connection header received
0619 18:56:16.442821 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:16.442827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:16.442940 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:16.443079 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:16.445316 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:16.446509 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:16.448601 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:16.449171 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:16.449332 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:16.453482 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:16.453490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:16.453494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:16.453542 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:16.457457 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:16.458863 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:16.458871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:16.458875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:16.459078 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:16.459754 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:16.459760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:16.459763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:16.460189 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:16.460361 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:16.461592 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0619 18:56:16.461884 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15146,"thread_start_us":16373,"threads_started":1}
I20260619 18:56:16.463840 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:16.423877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42901 (local address 127.15.91.2:41867)
0619 18:56:16.439748 (+ 15871us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:16.439833 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:16.442199 (+ 2366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:16.443651 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:16.443663 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:16.444266 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:16.444983 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:16.445008 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:16.446818 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:16.446824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:16.448344 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:16.448366 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:16.449475 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:16.449485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:16.449756 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:16.450603 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:16.450630 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:16.453185 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:16.457728 (+ 4543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:16.457744 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:16.457772 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:16.458601 (+ 829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:16.459239 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:16.459248 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:16.459255 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:16.459587 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:16.460402 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:16.460416 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:16.461008 (+ 592us) client_negotiation.cc:769] Sending connection context
0619 18:56:16.461396 (+ 388us) client_negotiation.cc:240] Negotiation successful
0619 18:56:16.462088 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15157,"thread_start_us":14773,"threads_started":1}
I20260619 18:56:16.466180 28442 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42901
I20260619 18:56:16.466646 28442 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:16.467783 28442 heartbeater.cc:511] Master 127.15.91.62:42901 requested a full tablet report, sending...
I20260619 18:56:16.470168 28086 ts_manager.cc:194] Registered new tserver with Master: d754e40fd4b245b7bd44aa79aaab41fb (127.15.91.2:34921)
I20260619 18:56:16.471616 28086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:41867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:56:16.490963 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:54383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42901
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:56:16.626303 28299 heartbeater.cc:503] Master 127.15.91.62:42901 was elected leader, sending a full tablet report...
W20260619 18:56:16.868068 28448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:16.868467 28448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:16.868643 28448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:16.876765 28448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:16.876927 28448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:16.877104 28448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:56:16.887861 28448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.91.20:54383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:42901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:16.889287 28448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:16.890887 28448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:16.905668 28453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:16.905711 28456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:16.905755 28454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:16.906817 28448 server_base.cc:1034] running on GCE node
I20260619 18:56:16.907636 28448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:16.908834 28448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:16.910132 28448 hybrid_clock.cc:648] HybridClock initialized: now 1781895376910059 us; error 55 us; skew 500 ppm
I20260619 18:56:16.910530 28448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:16 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895376, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:16.914149 28448 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:56:16.915963 28448 webserver.cc:460] Webserver started at http://127.15.91.3:46703/ using document root <none> and password file <none>
I20260619 18:56:16.916882 28448 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:16.917070 28448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:16.917490 28448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:56:16 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895376, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:56:16.933908 28448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "f49cb326899f46a293fcd64981aba767"
format_stamp: "Formatted at 2026-06-19 18:56:16 on dist-test-slave-dm3f"
server_key: "a2103423ec445a56d0313db3dedb57c1"
server_key_iv: "ea7fe280015ed07e4ed60678d87f3f46"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:16.935040 28448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "f49cb326899f46a293fcd64981aba767"
format_stamp: "Formatted at 2026-06-19 18:56:16 on dist-test-slave-dm3f"
server_key: "a2103423ec445a56d0313db3dedb57c1"
server_key_iv: "ea7fe280015ed07e4ed60678d87f3f46"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:16.943049 28448 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260619 18:56:16.960139 28463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:16.961916 28448 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:56:16.962184 28448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "f49cb326899f46a293fcd64981aba767"
format_stamp: "Formatted at 2026-06-19 18:56:16 on dist-test-slave-dm3f"
server_key: "a2103423ec445a56d0313db3dedb57c1"
server_key_iv: "ea7fe280015ed07e4ed60678d87f3f46"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:16.962461 28448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:16.980134 28448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:16.984515 28448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:16.984872 28448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:16.985954 28448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:16.987584 28448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:16.987710 28448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:16.987872 28448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:16.987947 28448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:17.026734 28448 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:43829
I20260619 18:56:17.026844 28575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:43829 every 8 connection(s)
I20260619 18:56:17.028358 28448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:56:17.030022 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 28448
I20260619 18:56:17.030216 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:56:17.030795 15724 external_mini_cluster.cc:1428] Setting key 883a1e09c66e707cfa1b1799f4f17deb
Jun 19 18:56:17 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895376, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:17.056070 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:17.031974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:54439 (local address 127.15.91.62:42901)
0619 18:56:17.032675 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:17.032683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:17.033983 (+ 1300us) server_negotiation.cc:407] Connection header received
0619 18:56:17.036166 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:17.036171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:17.036278 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:17.036390 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:17.038385 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:17.039295 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:17.042041 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:17.042642 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:17.042975 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:17.048482 (+ 5507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:17.048488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:17.048492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:17.048543 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:17.051931 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:17.053025 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:17.053031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:17.053033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:17.053209 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:17.053910 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:17.053914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:17.053916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:17.054304 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:17.054432 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:17.055473 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0619 18:56:17.055711 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":398,"threads_started":1}
I20260619 18:56:17.057157 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:17.032747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42901 (local address 127.15.91.3:54439)
0619 18:56:17.033776 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:17.033837 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:17.035800 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:17.036717 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:17.036727 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:17.037400 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:17.038124 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:17.038147 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:17.039629 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:17.039636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:17.041622 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:17.041653 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:17.043131 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:17.043142 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:17.043751 (+ 609us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:17.044763 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:17.044803 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:17.048178 (+ 3375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:17.052198 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:17.052210 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:17.052229 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:17.052845 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:17.053413 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:17.053417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:17.053420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:17.053781 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:17.054459 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:17.054467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:17.054946 (+ 479us) client_negotiation.cc:769] Sending connection context
0619 18:56:17.055248 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 18:56:17.055719 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":446,"threads_started":1}
I20260619 18:56:17.059049 28576 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42901
I20260619 18:56:17.059481 28576 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:17.060491 28576 heartbeater.cc:511] Master 127.15.91.62:42901 requested a full tablet report, sending...
I20260619 18:56:17.062944 28086 ts_manager.cc:194] Registered new tserver with Master: f49cb326899f46a293fcd64981aba767 (127.15.91.3:43829)
I20260619 18:56:17.063829 28086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:54439
I20260619 18:56:17.075181 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:56:17.076993 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 28166
Jun 19 18:56:17 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:56:17.098304 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:17.078367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.15.91.1:43253)
0619 18:56:17.079139 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:17.079147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:17.079175 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:56:17.079240 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:17.079246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:17.079466 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:17.079610 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:17.081351 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:17.083057 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:17.084770 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:17.085496 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:17.085649 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:17.091521 (+ 5872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:17.091545 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:17.091564 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:17.091625 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:17.094981 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:17.095988 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:17.095996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:17.095999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:17.096196 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:17.096653 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:17.096657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:17.096659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:17.097099 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:17.097384 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:17.097724 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 18:56:17.097970 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":413,"threads_started":1}
I20260619 18:56:17.099805 28294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:17.185499 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 28166
I20260619 18:56:17.202911 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 28305
Jun 19 18:56:17 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:56:17.221534 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:17.203902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.15.91.2:34921)
0619 18:56:17.204548 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:17.204555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:17.204585 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:56:17.204653 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:17.204660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:17.204888 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:17.205044 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:17.206421 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:17.207585 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:17.209204 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:17.209835 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:17.209959 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:17.214933 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:17.214950 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:17.214967 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:17.215028 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:17.218226 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:17.219269 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:17.219277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:17.219280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:17.219452 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:17.219910 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:17.219913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:17.219915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:17.220311 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:17.220597 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:17.220923 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 18:56:17.221196 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":331,"threads_started":1}
I20260619 18:56:17.223124 28437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:17.309258 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 28305
I20260619 18:56:17.330235 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 28448
Jun 19 18:56:17 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:56:17.351173 28585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:17.331877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.15.91.3:43829)
0619 18:56:17.332663 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:17.332674 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:17.332707 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:56:17.332774 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:17.332781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:17.333027 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:17.333227 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:17.335128 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:17.336231 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:17.337715 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:17.338345 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:17.338520 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:17.343857 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:17.343875 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:17.343894 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:17.343945 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:17.347687 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:17.348678 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:17.348687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:17.348690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:17.348865 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:17.349230 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:17.349233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:17.349235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:17.349712 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:17.349988 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:17.350413 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 18:56:17.350708 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":422,"threads_started":1}
I20260619 18:56:17.352713 28571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:17.442395 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 28448
I20260619 18:56:17.459270 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 28054
I20260619 18:56:17.460579 28116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:17.569583 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 28054
I20260619 18:56:17.596421 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:42901
--webserver_interface=127.15.91.62
--webserver_port=33237
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:42901
--ranger_config_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:54383/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.91.20:54383/kms with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:17.956753 28588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:17.957162 28588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:17.957333 28588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:17.965143 28588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:56:17.965345 28588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:17.965461 28588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:17.965595 28588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:56:17.965684 28588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:56:17.977398 28588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.91.20:54383/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:42901
--ranger_config_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:42901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=33237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:17.978961 28588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:17.980499 28588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:17.994381 28593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:17.994311 28594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:17.994359 28596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:17.995471 28588 server_base.cc:1034] running on GCE node
I20260619 18:56:17.996346 28588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:17.998128 28588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:17.999411 28588 hybrid_clock.cc:648] HybridClock initialized: now 1781895377999354 us; error 58 us; skew 500 ppm
I20260619 18:56:17.999966 28588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:17 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895377, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:18.003665 28588 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:56:18.005609 28588 webserver.cc:460] Webserver started at http://127.15.91.62:33237/ using document root <none> and password file <none>
I20260619 18:56:18.006431 28588 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:18.006611 28588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:56:18 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895377, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:56:18.097401 28588 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:56:18.100967 28604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:18.102811 28588 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:56:18.103070 28588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "85a857adf9bb436abbc2b14d9cd1a8d3"
format_stamp: "Formatted at 2026-06-19 18:56:13 on dist-test-slave-dm3f"
server_key: "e1d837c46d89019296c3fa12b80e5694"
server_key_iv: "456b204b861a1f3a2129d571ad3ca240"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:18.104188 28588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:18.135177 28588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:18.139559 28588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:18.139905 28588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:18.160887 28588 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:42901
I20260619 18:56:18.160910 28655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:42901 every 8 connection(s)
I20260619 18:56:18.162560 28588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:56:18.163564 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 28588
I20260619 18:56:18.164626 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:54383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:43253
--local_ip_for_outbound_sockets=127.15.91.1
--tserver_master_addrs=127.15.91.62:42901
--webserver_port=38081
--webserver_interface=127.15.91.1
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.91.20:54383/kms with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:56:18.170527 28656 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:56:18.176486 28656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3: Bootstrap starting.
I20260619 18:56:18.207616 28656 log.cc:824] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:18.221367 28656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:56:18.222383 28656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3: Bootstrap complete.
I20260619 18:56:18.234256 28656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } }
I20260619 18:56:18.235548 28656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a857adf9bb436abbc2b14d9cd1a8d3, State: Initialized, Role: FOLLOWER
I20260619 18:56:18.236359 28656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } }
I20260619 18:56:18.236696 28656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:56:18.236842 28656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:56:18.237035 28656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:56:18.241614 28656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } }
I20260619 18:56:18.242265 28656 leader_election.cc:302] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85a857adf9bb436abbc2b14d9cd1a8d3; no voters:
I20260619 18:56:18.243402 28656 leader_election.cc:288] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:56:18.243551 28660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:56:18.245219 28660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [term 2 LEADER]: Becoming Leader. State: Replica: 85a857adf9bb436abbc2b14d9cd1a8d3, State: Running, Role: LEADER
I20260619 18:56:18.246187 28656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:56:18.246049 28660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } }
I20260619 18:56:18.249500 28661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a857adf9bb436abbc2b14d9cd1a8d3. Latest consensus state: current_term: 2 leader_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } } }
I20260619 18:56:18.250128 28661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:18.249539 28662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a857adf9bb436abbc2b14d9cd1a8d3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42901 } } }
I20260619 18:56:18.250430 28662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:18.265421 28666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:56:18.275467 28666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:56:18.277403 28666 catalog_manager.cc:1259] Loaded cluster ID: ea39f97941194e409ce4ebec2d6dff95
I20260619 18:56:18.277655 28666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:56:18.282943 28666 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:56:18.284085 28656 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:56:18.285120 28666 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3: Loaded TSK: 0
I20260619 18:56:18.287200 28666 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:56:18.661818 28658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:18.662371 28658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:18.662582 28658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:18.674240 28658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:18.674429 28658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:18.674631 28658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:56:18.690470 28658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.91.20:54383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:43253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=38081
--tserver_master_addrs=127.15.91.62:42901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:18.692557 28658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:18.694581 28658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:18.733043 28693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:18.749179 28694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:18.782167 28658 server_base.cc:1034] running on GCE node
W20260619 18:56:18.783434 28696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:18.784582 28658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:18.786245 28658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:18.787585 28658 hybrid_clock.cc:648] HybridClock initialized: now 1781895378787510 us; error 78 us; skew 500 ppm
I20260619 18:56:18.788009 28658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:18 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895378, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:18.792068 28658 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:56:18.814771 28658 webserver.cc:460] Webserver started at http://127.15.91.1:38081/ using document root <none> and password file <none>
I20260619 18:56:18.816043 28658 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:18.816339 28658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:56:18 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895378, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:56:18.878031 28658 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.011s sys 0.004s
I20260619 18:56:18.885156 28704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:18.888257 28658 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.002s
I20260619 18:56:18.888501 28658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "d4467accee9549949396a3ab71bb9cea"
format_stamp: "Formatted at 2026-06-19 18:56:15 on dist-test-slave-dm3f"
server_key: "daec6e2a650ba3d126309c3ce622509d"
server_key_iv: "707a48d366154c67384f57ced8dde6b2"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:18.890162 28658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:18.916112 28658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:18.921294 28658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:18.921650 28658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:18.922880 28658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:18.924626 28658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:18.924772 28658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:18.924912 28658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:18.925007 28658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:18.991961 28658 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:43253
I20260619 18:56:18.992009 28817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:43253 every 8 connection(s)
I20260619 18:56:18.993804 28658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:56:18.996583 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 28658
I20260619 18:56:18.997479 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:54383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:34921
--local_ip_for_outbound_sockets=127.15.91.2
--tserver_master_addrs=127.15.91.62:42901
--webserver_port=40849
--webserver_interface=127.15.91.2
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.91.20:54383/kms with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:56:19 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895378, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:19.038946 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:19.001352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42901 (local address 127.15.91.1:46169)
0619 18:56:19.008861 (+ 7509us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:19.008928 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:19.012868 (+ 3940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:19.016140 (+ 3272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:19.016154 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:19.016921 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:19.017892 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:19.017930 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:19.020886 (+ 2956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:19.020893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:19.023295 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:19.023326 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:19.025242 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:19.025250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:19.025531 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:19.026368 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:19.026397 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:19.028987 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:19.033396 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:19.033407 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:19.033428 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:19.034041 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:19.034746 (+ 705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:19.034751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:19.034755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:19.035087 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:19.036011 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:19.036021 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:19.036398 (+ 377us) client_negotiation.cc:769] Sending connection context
0619 18:56:19.036722 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 18:56:19.037383 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7118,"thread_start_us":6923,"threads_started":1}
I20260619 18:56:19.038945 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:19.002756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:46169 (local address 127.15.91.62:42901)
0619 18:56:19.007368 (+ 4612us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:19.007395 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:19.009366 (+ 1971us) server_negotiation.cc:407] Connection header received
0619 18:56:19.013826 (+ 4460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:19.013852 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:19.014548 (+ 696us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:19.015104 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:19.018323 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:19.020504 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:19.023640 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:19.024519 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:19.024696 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:19.029388 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:19.029403 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:19.029426 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:19.029494 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:19.033161 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:19.034285 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:19.034294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:19.034301 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:19.034535 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:19.035248 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:19.035252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:19.035255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:19.035824 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:19.036066 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:19.036979 (+ 913us) server_negotiation.cc:299] Negotiation successful
0619 18:56:19.037406 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4220,"thread_start_us":3928,"threads_started":1}
I20260619 18:56:19.045055 28818 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42901
I20260619 18:56:19.045466 28818 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:19.046525 28818 heartbeater.cc:511] Master 127.15.91.62:42901 requested a full tablet report, sending...
W20260619 18:56:19.051081 28818 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:42901 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:56:19.051457 28818 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:19.051798 28818 heartbeater.cc:511] Master 127.15.91.62:42901 requested a full tablet report, sending...
I20260619 18:56:19.053323 28818 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:19.053666 28818 heartbeater.cc:511] Master 127.15.91.62:42901 requested a full tablet report, sending...
W20260619 18:56:19.055044 28818 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:56:19.501122 28822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:19.501497 28822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:19.501680 28822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:19.510799 28822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:19.510985 28822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:19.511193 28822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:56:19.523505 28822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.91.20:54383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:34921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=40849
--tserver_master_addrs=127.15.91.62:42901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:19.524991 28822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:19.526695 28822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:19.547245 28838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:19.546931 28834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:19.547333 28835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:19.548134 28822 server_base.cc:1034] running on GCE node
I20260619 18:56:19.548866 28822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:19.550124 28822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:19.551407 28822 hybrid_clock.cc:648] HybridClock initialized: now 1781895379551358 us; error 51 us; skew 500 ppm
I20260619 18:56:19.551808 28822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:19 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895379, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:19.556322 28822 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:56:19.558887 28822 webserver.cc:460] Webserver started at http://127.15.91.2:40849/ using document root <none> and password file <none>
I20260619 18:56:19.560077 28822 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:19.560335 28822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:56:19 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895379, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:56:19.590021 28822 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 18:56:19.593628 28847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:19.595564 28822 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:56:19.595862 28822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "d754e40fd4b245b7bd44aa79aaab41fb"
format_stamp: "Formatted at 2026-06-19 18:56:16 on dist-test-slave-dm3f"
server_key: "bdace31f8d52382120aa26cc59a3074f"
server_key_iv: "0cfb309d069bb8a70ef0701fb601b767"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:19.597216 28822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:19.636686 28822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:19.643456 28822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:19.644066 28822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:19.646613 28822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:19.648829 28822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:19.649016 28822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 18:56:19.649196 28822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:19.649282 28822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:19.742607 28822 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:34921
I20260619 18:56:19.742692 28961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:34921 every 8 connection(s)
Jun 19 18:56:19 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781895379, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:19.744939 28822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:56:19.754518 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 28822
I20260619 18:56:19.755901 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.91.20:54383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:43829
--local_ip_for_outbound_sockets=127.15.91.3
--tserver_master_addrs=127.15.91.62:42901
--webserver_port=46703
--webserver_interface=127.15.91.3
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.91.20:54383/kms with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:56:19 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895379, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:19.824000 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:19.750997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:41677 (local address 127.15.91.62:42901)
0619 18:56:19.782089 (+ 31092us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:19.782097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:19.786786 (+ 4689us) server_negotiation.cc:407] Connection header received
0619 18:56:19.786877 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:19.786883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:19.786998 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:19.787098 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:19.797483 (+ 10385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:19.799271 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:19.808684 (+ 9413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:19.809579 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:19.809742 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:19.815689 (+ 5947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:19.815697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:19.815703 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:19.815754 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:19.819587 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:19.820791 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:19.820797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:19.820801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:19.820992 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:19.821655 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:19.821661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:19.821665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:19.822061 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:19.822235 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:19.823278 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0619 18:56:19.823574 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30919,"thread_start_us":31106,"threads_started":1}
I20260619 18:56:19.825407 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:19.749085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42901 (local address 127.15.91.2:41677)
0619 18:56:19.783425 (+ 34340us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:19.783514 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:19.786247 (+ 2733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:19.791605 (+ 5358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:19.791627 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:19.792635 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:19.796915 (+ 4280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:19.797089 (+ 174us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:19.799644 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:19.799654 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:19.808358 (+ 8704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:19.808391 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:19.810247 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:19.810259 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:19.810692 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:19.811757 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:19.811801 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:19.815257 (+ 3456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:19.819832 (+ 4575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:19.819845 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:19.819868 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:19.820600 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:19.821137 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:19.821144 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:19.821148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:19.821466 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:19.822225 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:19.822235 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:19.822763 (+ 528us) client_negotiation.cc:769] Sending connection context
0619 18:56:19.823108 (+ 345us) client_negotiation.cc:240] Negotiation successful
0619 18:56:19.823748 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33936,"thread_start_us":40469,"threads_started":1}
I20260619 18:56:19.827327 28962 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42901
I20260619 18:56:19.827726 28962 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:19.828691 28962 heartbeater.cc:511] Master 127.15.91.62:42901 requested a full tablet report, sending...
I20260619 18:56:19.831248 28621 ts_manager.cc:194] Registered new tserver with Master: d754e40fd4b245b7bd44aa79aaab41fb (127.15.91.2:34921)
I20260619 18:56:19.833518 28621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:41677
I20260619 18:56:20.055244 28818 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:20.055837 28818 heartbeater.cc:511] Master 127.15.91.62:42901 requested a full tablet report, sending...
I20260619 18:56:20.057507 28621 ts_manager.cc:194] Registered new tserver with Master: d4467accee9549949396a3ab71bb9cea (127.15.91.1:43253)
I20260619 18:56:20.059132 28621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:46169
Jun 19 18:56:20 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895379, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
W20260619 18:56:20.295647 28966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:20.296236 28966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:20.296427 28966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:20.308833 28966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:20.309026 28966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:20.309222 28966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:56:20.326074 28966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.91.20:54383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:43829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=46703
--tserver_master_addrs=127.15.91.62:42901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:20.328474 28966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:20.330601 28966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:20.350925 28972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:20.354399 28973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:20.368673 28976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:20.368981 28966 server_base.cc:1034] running on GCE node
I20260619 18:56:20.370030 28966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:20.371970 28966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:20.373344 28966 hybrid_clock.cc:648] HybridClock initialized: now 1781895380373200 us; error 148 us; skew 500 ppm
I20260619 18:56:20.373800 28966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:20 dist-test-slave-dm3f krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895380, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:20.378588 28966 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:56:20.380913 28966 webserver.cc:460] Webserver started at http://127.15.91.3:46703/ using document root <none> and password file <none>
I20260619 18:56:20.381983 28966 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:20.382175 28966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:56:20 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895380, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:56:20.406370 28966 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:56:20.410596 28983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:20.412668 28966 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:56:20.412918 28966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "f49cb326899f46a293fcd64981aba767"
format_stamp: "Formatted at 2026-06-19 18:56:16 on dist-test-slave-dm3f"
server_key: "a2103423ec445a56d0313db3dedb57c1"
server_key_iv: "ea7fe280015ed07e4ed60678d87f3f46"
server_key_version: "kuduclusterkey@0"
I20260619 18:56:20.414517 28966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:20.434276 28966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:20.439878 28966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:20.440227 28966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:20.441461 28966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:20.443204 28966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:20.443342 28966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:20.443492 28966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:20.443580 28966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:20.489777 28966 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:43829
I20260619 18:56:20.489823 29095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:43829 every 8 connection(s)
I20260619 18:56:20.491640 28966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:56:20.497139 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 28966
Jun 19 18:56:20 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895380, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:20.547505 29101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.507550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:34163 (local address 127.15.91.62:42901)
0619 18:56:20.518789 (+ 11239us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:20.518795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:20.521008 (+ 2213us) server_negotiation.cc:407] Connection header received
0619 18:56:20.524187 (+ 3179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:20.524193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:20.524318 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:20.524435 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:20.527314 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.528598 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.531568 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.532331 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.532469 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.537617 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:20.537629 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:20.537633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:20.537689 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:20.542663 (+ 4974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:20.543780 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:20.543786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:20.543790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:20.544071 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:20.544753 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:20.544758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:20.544762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:20.545165 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:20.545321 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:20.546646 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0619 18:56:20.546998 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11049,"spinlock_wait_cycles":6400,"thread_start_us":11247,"threads_started":1}
I20260619 18:56:20.548532 29100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.506748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:42901 (local address 127.15.91.3:34163)
0619 18:56:20.520749 (+ 14001us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:20.520829 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:20.523562 (+ 2733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:20.525212 (+ 1650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:20.525231 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:20.525925 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:20.526930 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.526975 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.528880 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.528887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.531279 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.531308 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.532976 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.532984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.533386 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.534181 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:20.534227 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:20.537222 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:20.542882 (+ 5660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:20.542895 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:20.542916 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:20.543433 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:20.544248 (+ 815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:20.544254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:20.544258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:20.544589 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:20.545411 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:20.545421 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:20.545948 (+ 527us) client_negotiation.cc:769] Sending connection context
0619 18:56:20.546288 (+ 340us) client_negotiation.cc:240] Negotiation successful
0619 18:56:20.547003 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13572,"thread_start_us":13282,"threads_started":1}
I20260619 18:56:20.550906 29096 heartbeater.cc:347] Connected to a master server at 127.15.91.62:42901
I20260619 18:56:20.551004 29099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.499121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.15.91.62:42901)
0619 18:56:20.516673 (+ 17552us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:20.516680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:20.516706 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:56:20.516781 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:20.516786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:20.516888 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:20.517038 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:20.521283 (+ 4245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.522331 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.527426 (+ 5095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.528286 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.528463 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.537695 (+ 9232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:20.537704 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:20.537709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:20.537788 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:20.542396 (+ 4608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:20.543780 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:20.543786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:20.543789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:20.544050 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:20.548745 (+ 4695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:20.548753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:20.548757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:20.549175 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:20.549494 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:20.550249 (+ 755us) server_negotiation.cc:299] Negotiation successful
0619 18:56:20.550559 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17401,"spinlock_wait_cycles":7552,"thread_start_us":17230,"threads_started":1}
I20260619 18:56:20.551337 29096 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:20.552315 29096 heartbeater.cc:511] Master 127.15.91.62:42901 requested a full tablet report, sending...
I20260619 18:56:20.554273 28621 ts_manager.cc:194] Registered new tserver with Master: f49cb326899f46a293fcd64981aba767 (127.15.91.3:43829)
I20260619 18:56:20.555567 28621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:34163
I20260619 18:56:20.556350 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:56:20.585296 29099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.562641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.15.91.62:42901)
0619 18:56:20.563152 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:20.563159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:20.566008 (+ 2849us) server_negotiation.cc:407] Connection header received
0619 18:56:20.566375 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:20.566382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:20.566508 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:20.566591 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:20.569202 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.570276 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.572579 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.573287 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.573479 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.577375 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:20.577382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:20.577387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:20.577443 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:20.581270 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:20.582493 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:20.582521 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:20.582525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:20.582755 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:20.583328 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:20.583334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:20.583337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:20.583756 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:20.584007 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:20.584684 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 18:56:20.584963 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260619 18:56:20.593133 28621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:56:20.605193 28621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Jun 19 18:56:20 dist-test-slave-dm3f krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895379, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for HTTP/127.15.91.20@KRBTEST.COM
I20260619 18:56:20.676584 29112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.644861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:43829 (local address 127.0.0.1:36854)
0619 18:56:20.664333 (+ 19472us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:20.664390 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:20.664642 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:20.667020 (+ 2378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:20.667030 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:20.667094 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:20.667576 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.667594 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.669958 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.669966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.672577 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.672609 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.675391 (+ 2782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.675399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.675668 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.675752 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 18:56:20.675935 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 18:56:20.676169 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19079,"thread_start_us":20028,"threads_started":1}
I20260619 18:56:20.677665 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.645529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.15.91.3:43829)
0619 18:56:20.662892 (+ 17363us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:20.662903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:20.665714 (+ 2811us) server_negotiation.cc:407] Connection header received
0619 18:56:20.665844 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:20.665852 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:20.666181 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:20.666419 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:20.667871 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.669707 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.672922 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.674642 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.674893 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.675176 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:20.677078 (+ 1902us) server_negotiation.cc:299] Negotiation successful
0619 18:56:20.677311 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17097,"thread_start_us":17430,"threads_started":1}
I20260619 18:56:20.684621 29031 tablet_service.cc:1467] Processing CreateTablet for tablet 77af241510ce4c828c9dc79a96887c36 (DEFAULT_TABLE table=test-table [id=8688c837df4e40028669586a4a434c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:20.687918 29031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77af241510ce4c828c9dc79a96887c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:20.698297 29113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.645148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:34921 (local address 127.0.0.1:43724)
0619 18:56:20.680915 (+ 35767us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:20.680943 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:20.681154 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:20.683850 (+ 2696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:20.683856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:20.683893 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:20.684264 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.684281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.689299 (+ 5018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.689306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.691493 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.691521 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.697310 (+ 5789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.697319 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.697544 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.697586 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 18:56:20.697717 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 18:56:20.697888 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35600,"thread_start_us":39917,"threads_started":1}
I20260619 18:56:20.699930 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.645694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.15.91.2:34921)
0619 18:56:20.663603 (+ 17909us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:20.663613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:20.682815 (+ 19202us) server_negotiation.cc:407] Connection header received
0619 18:56:20.682958 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:20.682967 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:20.683252 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:20.683495 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:20.685669 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.688981 (+ 3312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.694892 (+ 5911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.696633 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.696828 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.697058 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:20.699329 (+ 2271us) server_negotiation.cc:299] Negotiation successful
0619 18:56:20.699558 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17718,"thread_start_us":19973,"threads_started":1}
I20260619 18:56:20.715363 28897 tablet_service.cc:1467] Processing CreateTablet for tablet 77af241510ce4c828c9dc79a96887c36 (DEFAULT_TABLE table=test-table [id=8688c837df4e40028669586a4a434c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:20.717492 28897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77af241510ce4c828c9dc79a96887c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:20.719970 29113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.691368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:43253 (local address 127.0.0.1:53660)
0619 18:56:20.698908 (+ 7540us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:20.698932 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:20.699093 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:20.703930 (+ 4837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:20.703936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:20.703975 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:20.704185 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.704198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.706466 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.706473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.708817 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.708894 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.718923 (+ 10029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.718935 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.719215 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.719265 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 18:56:20.719405 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 18:56:20.719599 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7426,"thread_start_us":17236,"threads_started":1}
I20260619 18:56:20.720984 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.685918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.15.91.1:43253)
0619 18:56:20.691271 (+ 5353us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:20.691281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:20.702749 (+ 11468us) server_negotiation.cc:407] Connection header received
0619 18:56:20.702911 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:20.702920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:20.703317 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:20.703601 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:20.704426 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.706200 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.709162 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.710955 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.711133 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.711408 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:20.720402 (+ 8994us) server_negotiation.cc:299] Negotiation successful
0619 18:56:20.720651 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5143,"thread_start_us":4927,"threads_started":1}
I20260619 18:56:20.732749 28753 tablet_service.cc:1467] Processing CreateTablet for tablet 77af241510ce4c828c9dc79a96887c36 (DEFAULT_TABLE table=test-table [id=8688c837df4e40028669586a4a434c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:20.735363 28753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77af241510ce4c828c9dc79a96887c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:20.743597 29118 tablet_bootstrap.cc:492] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: Bootstrap starting.
I20260619 18:56:20.747969 29118 tablet_bootstrap.cc:654] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:20.749671 29118 log.cc:824] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:20.759349 29118 tablet_bootstrap.cc:492] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: No bootstrap required, opened a new log
I20260619 18:56:20.759720 29118 ts_tablet_manager.cc:1397] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260619 18:56:20.775115 29120 tablet_bootstrap.cc:492] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb: Bootstrap starting.
I20260619 18:56:20.777290 29118 raft_consensus.cc:348] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49cb326899f46a293fcd64981aba767" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43829 } } peers { permanent_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34921 } } peers { permanent_uuid: "d4467accee9549949396a3ab71bb9cea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 43253 } }
I20260619 18:56:20.778687 29118 raft_consensus.cc:374] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:20.779008 29118 raft_consensus.cc:729] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49cb326899f46a293fcd64981aba767, State: Initialized, Role: FOLLOWER
I20260619 18:56:20.780386 29118 consensus_queue.cc:260] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49cb326899f46a293fcd64981aba767" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43829 } } peers { permanent_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34921 } } peers { permanent_uuid: "d4467accee9549949396a3ab71bb9cea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 43253 } }
I20260619 18:56:20.791039 29096 heartbeater.cc:503] Master 127.15.91.62:42901 was elected leader, sending a full tablet report...
I20260619 18:56:20.792284 29118 ts_tablet_manager.cc:1428] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: Time spent starting tablet: real 0.032s user 0.020s sys 0.004s
I20260619 18:56:20.793015 29120 tablet_bootstrap.cc:654] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:20.794592 29120 log.cc:824] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:20.795157 29122 tablet_bootstrap.cc:492] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: Bootstrap starting.
I20260619 18:56:20.801059 29122 tablet_bootstrap.cc:654] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:20.802645 29122 log.cc:824] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:20.805039 29120 tablet_bootstrap.cc:492] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb: No bootstrap required, opened a new log
I20260619 18:56:20.805485 29120 ts_tablet_manager.cc:1397] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.000s
I20260619 18:56:20.806629 29122 tablet_bootstrap.cc:492] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: No bootstrap required, opened a new log
I20260619 18:56:20.807052 29122 ts_tablet_manager.cc:1397] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260619 18:56:20.817899 29120 raft_consensus.cc:348] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49cb326899f46a293fcd64981aba767" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43829 } } peers { permanent_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34921 } } peers { permanent_uuid: "d4467accee9549949396a3ab71bb9cea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 43253 } }
I20260619 18:56:20.818782 29120 raft_consensus.cc:374] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:20.819031 29120 raft_consensus.cc:729] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d754e40fd4b245b7bd44aa79aaab41fb, State: Initialized, Role: FOLLOWER
I20260619 18:56:20.820124 29120 consensus_queue.cc:260] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49cb326899f46a293fcd64981aba767" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43829 } } peers { permanent_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34921 } } peers { permanent_uuid: "d4467accee9549949396a3ab71bb9cea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 43253 } }
I20260619 18:56:20.820217 29122 raft_consensus.cc:348] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49cb326899f46a293fcd64981aba767" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43829 } } peers { permanent_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34921 } } peers { permanent_uuid: "d4467accee9549949396a3ab71bb9cea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 43253 } }
I20260619 18:56:20.820839 29122 raft_consensus.cc:374] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:20.820999 29122 raft_consensus.cc:729] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4467accee9549949396a3ab71bb9cea, State: Initialized, Role: FOLLOWER
I20260619 18:56:20.822084 29122 consensus_queue.cc:260] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49cb326899f46a293fcd64981aba767" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43829 } } peers { permanent_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34921 } } peers { permanent_uuid: "d4467accee9549949396a3ab71bb9cea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 43253 } }
I20260619 18:56:20.837493 28962 heartbeater.cc:503] Master 127.15.91.62:42901 was elected leader, sending a full tablet report...
I20260619 18:56:20.839411 29120 ts_tablet_manager.cc:1428] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb: Time spent starting tablet: real 0.034s user 0.015s sys 0.007s
I20260619 18:56:20.851657 28818 heartbeater.cc:503] Master 127.15.91.62:42901 was elected leader, sending a full tablet report...
I20260619 18:56:20.852474 29122 ts_tablet_manager.cc:1428] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: Time spent starting tablet: real 0.045s user 0.010s sys 0.030s
I20260619 18:56:20.871229 29125 raft_consensus.cc:482] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:56:20.871616 29125 raft_consensus.cc:504] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49cb326899f46a293fcd64981aba767" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43829 } } peers { permanent_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34921 } } peers { permanent_uuid: "d4467accee9549949396a3ab71bb9cea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 43253 } }
I20260619 18:56:20.874130 29125 leader_election.cc:288] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f49cb326899f46a293fcd64981aba767 (127.15.91.3:43829), d4467accee9549949396a3ab71bb9cea (127.15.91.1:43253)
I20260619 18:56:20.900520 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.876306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:36659 (local address 127.15.91.1:43253)
0619 18:56:20.876547 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:20.876553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:20.884189 (+ 7636us) server_negotiation.cc:407] Connection header received
0619 18:56:20.885942 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:20.885950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:20.886074 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:20.886151 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:20.890325 (+ 4174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.891704 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.897075 (+ 5371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.898462 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.898620 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.899116 (+ 496us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:20.900040 (+ 924us) server_negotiation.cc:299] Negotiation successful
0619 18:56:20.900236 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:56:20.903488 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.875633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:43253 (local address 127.15.91.2:36659)
0619 18:56:20.884041 (+ 8408us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:20.884064 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:20.884556 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:20.886401 (+ 1845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:20.886407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:20.886440 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:20.886891 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.886907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.891962 (+ 5055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.891968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.896749 (+ 4781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.896765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.899413 (+ 2648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.899422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.899626 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.899669 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 18:56:20.902883 (+ 3214us) client_negotiation.cc:240] Negotiation successful
0619 18:56:20.903070 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8255,"thread_start_us":11531,"threads_started":1}
I20260619 18:56:20.905030 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.874790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:51769 (local address 127.15.91.3:43829)
0619 18:56:20.875030 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:20.875037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:20.883537 (+ 8500us) server_negotiation.cc:407] Connection header received
0619 18:56:20.884783 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:20.884793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:20.884887 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:20.884962 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:20.887171 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.888493 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.901226 (+ 12733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.902541 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.902682 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.902824 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:20.904591 (+ 1767us) server_negotiation.cc:299] Negotiation successful
0619 18:56:20.904761 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:56:20.906028 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.874490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:43829 (local address 127.15.91.2:51769)
0619 18:56:20.883369 (+ 8879us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:20.883396 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:20.884362 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:20.885221 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:20.885227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:20.885263 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:20.885701 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.885716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.888752 (+ 3036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.888761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.896650 (+ 7889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:20.896682 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.904093 (+ 7411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.904101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:20.904331 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.904372 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 18:56:20.905505 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0619 18:56:20.905682 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8706,"thread_start_us":8471,"threads_started":1}
I20260619 18:56:20.907428 28773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77af241510ce4c828c9dc79a96887c36" candidate_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4467accee9549949396a3ab71bb9cea" is_pre_election: true
I20260619 18:56:20.907593 29051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77af241510ce4c828c9dc79a96887c36" candidate_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49cb326899f46a293fcd64981aba767" is_pre_election: true
I20260619 18:56:20.908169 29051 raft_consensus.cc:2454] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d754e40fd4b245b7bd44aa79aaab41fb in term 0.
I20260619 18:56:20.908170 28773 raft_consensus.cc:2454] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d754e40fd4b245b7bd44aa79aaab41fb in term 0.
I20260619 18:56:20.909133 28850 leader_election.cc:302] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d754e40fd4b245b7bd44aa79aaab41fb, f49cb326899f46a293fcd64981aba767; no voters:
I20260619 18:56:20.909648 29125 raft_consensus.cc:2781] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:56:20.909824 29125 raft_consensus.cc:482] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:56:20.909930 29125 raft_consensus.cc:3037] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:20.913672 29125 raft_consensus.cc:504] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49cb326899f46a293fcd64981aba767" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43829 } } peers { permanent_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34921 } } peers { permanent_uuid: "d4467accee9549949396a3ab71bb9cea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 43253 } }
I20260619 18:56:20.915373 29051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77af241510ce4c828c9dc79a96887c36" candidate_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49cb326899f46a293fcd64981aba767"
I20260619 18:56:20.915647 29051 raft_consensus.cc:3037] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:20.919445 29051 raft_consensus.cc:2454] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d754e40fd4b245b7bd44aa79aaab41fb in term 1.
I20260619 18:56:20.920194 28850 leader_election.cc:302] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d754e40fd4b245b7bd44aa79aaab41fb, f49cb326899f46a293fcd64981aba767; no voters:
I20260619 18:56:20.921007 29129 raft_consensus.cc:2781] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:56:20.921954 28773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77af241510ce4c828c9dc79a96887c36" candidate_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4467accee9549949396a3ab71bb9cea"
I20260619 18:56:20.922271 28773 raft_consensus.cc:3037] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:20.925989 28773 raft_consensus.cc:2454] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d754e40fd4b245b7bd44aa79aaab41fb in term 1.
I20260619 18:56:20.926911 29125 leader_election.cc:288] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [CANDIDATE]: Term 1 election: Requested vote from peers f49cb326899f46a293fcd64981aba767 (127.15.91.3:43829), d4467accee9549949396a3ab71bb9cea (127.15.91.1:43253)
I20260619 18:56:20.928120 29129 raft_consensus.cc:686] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 1 LEADER]: Becoming Leader. State: Replica: d754e40fd4b245b7bd44aa79aaab41fb, State: Running, Role: LEADER
I20260619 18:56:20.928853 29129 consensus_queue.cc:237] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49cb326899f46a293fcd64981aba767" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43829 } } peers { permanent_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34921 } } peers { permanent_uuid: "d4467accee9549949396a3ab71bb9cea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 43253 } }
I20260619 18:56:20.938773 28620 catalog_manager.cc:5495] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb reported cstate change: term changed from 0 to 1, leader changed from <none> to d754e40fd4b245b7bd44aa79aaab41fb (127.15.91.2). New cstate: current_term: 1 leader_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49cb326899f46a293fcd64981aba767" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d754e40fd4b245b7bd44aa79aaab41fb" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4467accee9549949396a3ab71bb9cea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 43253 } health_report { overall_health: UNKNOWN } } }
I20260619 18:56:20.992388 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:20.985303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.15.91.2:34921)
0619 18:56:20.985511 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:20.985518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:20.985713 (+ 195us) server_negotiation.cc:407] Connection header received
0619 18:56:20.985981 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:20.985987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:20.986148 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:20.986323 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:56:20.987060 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.988055 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.989614 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:20.990038 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:20.990193 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:20.990797 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:56:20.991622 (+ 825us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:56:20.991789 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:20.991984 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 18:56:20.992087 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 18:56:21.000517 28819 tablet.cc:2369] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:56:21.004784 28773 raft_consensus.cc:1261] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea [term 1 FOLLOWER]: Refusing update from remote peer d754e40fd4b245b7bd44aa79aaab41fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:56:21.004832 29051 raft_consensus.cc:1261] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767 [term 1 FOLLOWER]: Refusing update from remote peer d754e40fd4b245b7bd44aa79aaab41fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:56:21.006181 29130 consensus_queue.cc:1035] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4467accee9549949396a3ab71bb9cea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 43253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:56:21.006860 29129 consensus_queue.cc:1035] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49cb326899f46a293fcd64981aba767" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:56:21.014829 29097 tablet.cc:2369] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:56:21.028636 29140 mvcc.cc:205] Tried to move back new op lower bound from 7298643480582705152 to 7298643480290668544. Current Snapshot: MvccSnapshot[applied={T|T < 7298643480582705152}]
I20260619 18:56:21.063498 28621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:56:21.063824 28621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:56:21.068787 28621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85a857adf9bb436abbc2b14d9cd1a8d3: Sending DeleteTablet for 3 replicas of tablet 77af241510ce4c828c9dc79a96887c36
I20260619 18:56:21.070194 29031 tablet_service.cc:1514] Processing DeleteTablet for tablet 77af241510ce4c828c9dc79a96887c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:21 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:36854
I20260619 18:56:21.071238 29149 tablet_replica.cc:330] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: stopping tablet replica
I20260619 18:56:21.071848 29149 raft_consensus.cc:2229] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:56:21.072505 29149 raft_consensus.cc:2258] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:21.072445 28897 tablet_service.cc:1514] Processing DeleteTablet for tablet 77af241510ce4c828c9dc79a96887c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:21 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:43724
I20260619 18:56:21.073307 28753 tablet_service.cc:1514] Processing DeleteTablet for tablet 77af241510ce4c828c9dc79a96887c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:21 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:53660
I20260619 18:56:21.074299 29151 tablet_replica.cc:330] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: stopping tablet replica
I20260619 18:56:21.074892 29151 raft_consensus.cc:2229] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:56:21.075520 29151 raft_consensus.cc:2258] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:21.077689 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 28658
I20260619 18:56:21.078163 29151 ts_tablet_manager.cc:1905] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:21.086521 29151 ts_tablet_manager.cc:1918] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:56:21.086918 29151 log.cc:1196] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/77af241510ce4c828c9dc79a96887c36
I20260619 18:56:21.087608 29151 ts_tablet_manager.cc:1939] T 77af241510ce4c828c9dc79a96887c36 P d4467accee9549949396a3ab71bb9cea: Deleting consensus metadata
I20260619 18:56:21.089924 28608 catalog_manager.cc:4841] TS d4467accee9549949396a3ab71bb9cea (127.15.91.1:43253): tablet 77af241510ce4c828c9dc79a96887c36 (table test-table [id=8688c837df4e40028669586a4a434c50]) successfully deleted
I20260619 18:56:21.093189 29149 ts_tablet_manager.cc:1905] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:21.101501 29149 ts_tablet_manager.cc:1918] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:56:21.102195 29149 log.cc:1196] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/77af241510ce4c828c9dc79a96887c36
I20260619 18:56:21.103065 29149 ts_tablet_manager.cc:1939] T 77af241510ce4c828c9dc79a96887c36 P f49cb326899f46a293fcd64981aba767: Deleting consensus metadata
I20260619 18:56:21.105727 28607 catalog_manager.cc:4841] TS f49cb326899f46a293fcd64981aba767 (127.15.91.3:43829): tablet 77af241510ce4c828c9dc79a96887c36 (table test-table [id=8688c837df4e40028669586a4a434c50]) successfully deleted
I20260619 18:56:21.109882 29150 tablet_replica.cc:330] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb: stopping tablet replica
I20260619 18:56:21.110456 29150 raft_consensus.cc:2229] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:56:21.112464 29150 raft_consensus.cc:2258] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:21.116384 29150 ts_tablet_manager.cc:1905] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:21.127650 29150 ts_tablet_manager.cc:1918] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:56:21.128095 29150 log.cc:1196] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/77af241510ce4c828c9dc79a96887c36
I20260619 18:56:21.128784 29150 ts_tablet_manager.cc:1939] T 77af241510ce4c828c9dc79a96887c36 P d754e40fd4b245b7bd44aa79aaab41fb: Deleting consensus metadata
I20260619 18:56:21.130826 28608 catalog_manager.cc:4841] TS d754e40fd4b245b7bd44aa79aaab41fb (127.15.91.2:34921): tablet 77af241510ce4c828c9dc79a96887c36 (table test-table [id=8688c837df4e40028669586a4a434c50]) successfully deleted
I20260619 18:56:21.132068 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:21.078892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.15.91.1:43253)
0619 18:56:21.079277 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:21.079285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:21.107183 (+ 27898us) server_negotiation.cc:407] Connection header received
0619 18:56:21.107551 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:21.107558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:21.107667 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:21.107749 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:21.111196 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:21.112037 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:21.114587 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:21.115313 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:21.115483 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:21.119181 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:21.119203 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:21.119225 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:21.119289 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:21.124002 (+ 4713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:21.128001 (+ 3999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:21.128016 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:21.128020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:21.128254 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:21.128849 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:21.128855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:21.128859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:21.129412 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:21.129795 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:21.131379 (+ 1584us) server_negotiation.cc:299] Negotiation successful
0619 18:56:21.131663 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260619 18:56:21.132897 28813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:21.283174 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 28658
I20260619 18:56:21.309356 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 28822
I20260619 18:56:21.333630 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:21.310247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.15.91.2:34921)
0619 18:56:21.310455 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:21.310461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:21.310629 (+ 168us) server_negotiation.cc:407] Connection header received
0619 18:56:21.310881 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:21.310887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:21.310983 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:21.311105 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:21.313137 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:21.314398 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:21.316679 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:21.317361 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:21.317546 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:21.324806 (+ 7260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:21.324828 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:21.324849 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:21.324933 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:21.329581 (+ 4648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:21.330828 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:21.330840 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:21.330844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:21.331086 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:21.331600 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:21.331605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:21.331608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:21.332106 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:21.332482 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:21.332914 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 18:56:21.333180 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:56:21.334517 28957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:21.463725 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 28822
I20260619 18:56:21.486917 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 28966
I20260619 18:56:21.503887 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:21.487898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.15.91.3:43829)
0619 18:56:21.488725 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:21.488730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:21.488748 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:56:21.488801 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:21.488805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:21.488866 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:21.489005 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:21.490504 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:21.491503 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:21.493017 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:21.493523 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:21.493690 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:21.496780 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:21.496795 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:21.496812 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:21.496874 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:21.500368 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:21.501325 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:21.501334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:21.501336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:21.501486 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:21.501958 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:21.501964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:21.501969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:21.502414 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:21.502837 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:21.503164 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 18:56:21.503435 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":601,"threads_started":1}
I20260619 18:56:21.504712 29091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:21.600626 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 28966
I20260619 18:56:21.619706 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 28588
I20260619 18:56:21.621148 28651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:21.737540 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 28588
I20260619 18:56:21.762578 15724 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 18:56:22.099061 15724 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 18:56:22.099301 15724 mini_ranger.cc:66] Stopping Ranger...
I20260619 18:56:22.445932 15724 mini_ranger.cc:68] Stopped Ranger
2026-06-19 18:56:22.446 UTC [27640] LOG: received smart shutdown request
2026-06-19 18:56:22.449 UTC [27640] LOG: background worker "logical replication launcher" (PID 27649) exited with exit code 1
2026-06-19 18:56:22.449 UTC [27644] LOG: shutting down
2026-06-19 18:56:22.599 UTC [27640] LOG: database system is shut down
2026-06-19T18:56:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (59547 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:56:22 dist-test-slave-dm3f krb5kdc[29166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:56:22 dist-test-slave-dm3f krb5kdc[29166](info): set up 2 sockets
Jun 19 18:56:22 dist-test-slave-dm3f krb5kdc[29166](info): commencing operation
krb5kdc: starting...
W20260619 18:56:23.847504 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:56:24 dist-test-slave-dm3f krb5kdc[29166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:56:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:56:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:56:24.090600 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46489
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46489,127.15.91.61:33667,127.15.91.60:40183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:24.478536 29179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:24.479022 29179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:24.479203 29179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:24.487241 29179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:56:24.487417 29179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:24.487538 29179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:24.487643 29179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:56:24.487721 29179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:56:24.498525 29179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46489,127.15.91.61:33667,127.15.91.60:40183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:24.500159 29179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:24.501726 29179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:24.516101 29184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:24.516125 29187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:24.516022 29185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:24.517066 29179 server_base.cc:1034] running on GCE node
I20260619 18:56:24.517987 29179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:24.519825 29179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:24.521147 29179 hybrid_clock.cc:648] HybridClock initialized: now 1781895384521101 us; error 70 us; skew 500 ppm
I20260619 18:56:24.521629 29179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:24 dist-test-slave-dm3f krb5kdc[29166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:24.525338 29179 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:56:24.527452 29179 webserver.cc:460] Webserver started at http://127.15.91.62:40197/ using document root <none> and password file <none>
I20260619 18:56:24.528290 29179 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:24.528429 29179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:24.528720 29179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:24.532598 29179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "82fc33347e1749d8acbc2438869742c1"
format_stamp: "Formatted at 2026-06-19 18:56:24 on dist-test-slave-dm3f"
server_key: "028473759d79d6e5a58aacee6bcec1a5"
server_key_iv: "de4587a959c4d23d2d82300b4ad6e277"
server_key_version: "encryptionkey@0"
I20260619 18:56:24.533612 29179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "82fc33347e1749d8acbc2438869742c1"
format_stamp: "Formatted at 2026-06-19 18:56:24 on dist-test-slave-dm3f"
server_key: "028473759d79d6e5a58aacee6bcec1a5"
server_key_iv: "de4587a959c4d23d2d82300b4ad6e277"
server_key_version: "encryptionkey@0"
I20260619 18:56:24.540225 29179 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:56:24.544701 29194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:24.546566 29179 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260619 18:56:24.546865 29179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "82fc33347e1749d8acbc2438869742c1"
format_stamp: "Formatted at 2026-06-19 18:56:24 on dist-test-slave-dm3f"
server_key: "028473759d79d6e5a58aacee6bcec1a5"
server_key_iv: "de4587a959c4d23d2d82300b4ad6e277"
server_key_version: "encryptionkey@0"
I20260619 18:56:24.547211 29179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:24.566267 29179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:24.571398 29179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:24.571784 29179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:24.594049 29179 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:46489
I20260619 18:56:24.594060 29245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:46489 every 8 connection(s)
I20260619 18:56:24.595728 29179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:56:24.598531 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 29179
I20260619 18:56:24.598840 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:56:24.599565 15724 external_mini_cluster.cc:1428] Setting key 28ae595fb753fccf8fa086c441e4eb8f
I20260619 18:56:24.601460 29246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:24.611829 29246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.91.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.61@KRBTEST.COM" created.
Jun 19 18:56:24 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:24.644050 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:24.616799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46489 (local address 127.0.0.1:52646)
0619 18:56:24.617946 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:24.618027 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:24.621454 (+ 3427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:24.622708 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:24.622749 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:24.623571 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:24.624094 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:24.624122 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:24.626197 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:24.626207 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:24.628464 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:24.628526 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:24.629881 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:24.629893 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:24.630173 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:24.631161 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:24.631208 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:24.634512 (+ 3304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:24.639169 (+ 4657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:24.639185 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:24.639200 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:24.639698 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:24.640196 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:24.640203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:24.640206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:24.640392 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:24.641287 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:24.641296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:24.641526 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 18:56:24.641800 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 18:56:24.642465 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":186,"thread_start_us":427,"threads_started":1}
I20260619 18:56:24.644011 29250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:24.618381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.15.91.62:46489)
0619 18:56:24.619253 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:24.619261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:24.619310 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:56:24.622038 (+ 2728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:24.622045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:24.622369 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:24.622582 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:24.624396 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:24.625964 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:24.628803 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:24.629503 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:24.629710 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:24.634927 (+ 5217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:24.634938 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:24.634953 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:24.634996 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:24.638969 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:24.639879 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:24.639893 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:24.639895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:24.640072 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:24.640534 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:24.640539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:24.640541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:24.641132 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:24.641370 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:24.641940 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 18:56:24.642340 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":484,"threads_started":1}
I20260619 18:56:24.648187 29246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:56:24.649670 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:24.649126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60302)
0619 18:56:24.649372 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:24.649530 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 18:56:24.650060 29196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
I20260619 18:56:24.652320 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:24.651765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60306)
0619 18:56:24.651979 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:24.652070 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 18:56:24.652619 29246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:33667: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab.
Entry for principal kudu/127.15.91.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab.
I20260619 18:56:24.659514 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.61:33667
--webserver_interface=127.15.91.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab
--principal=kudu/127.15.91.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46489,127.15.91.61:33667,127.15.91.60:40183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:56:24.702000 29246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } attempt: 1
I20260619 18:56:24.703296 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:24.702834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60320)
0619 18:56:24.703025 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:24.703147 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 18:56:24.704548 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:24.704114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60334)
0619 18:56:24.704289 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:24.704384 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 18:56:24.704794 29246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:33667: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
I20260619 18:56:24.773116 29246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } attempt: 2
I20260619 18:56:24.774995 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:24.774297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60350)
0619 18:56:24.774569 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:24.774674 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 18:56:24.776575 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:24.776110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60356)
0619 18:56:24.776272 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:24.776388 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 18:56:24.776788 29246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:33667: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
I20260619 18:56:24.868093 29246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } attempt: 3
I20260619 18:56:24.869519 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:24.868941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60360)
0619 18:56:24.869165 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:24.869319 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 18:56:24.870694 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:24.870267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60364)
0619 18:56:24.870432 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:24.870521 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 18:56:24.870961 29246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:33667: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
I20260619 18:56:25.014257 29246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } attempt: 4
I20260619 18:56:25.015688 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.014994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60368)
0619 18:56:25.015374 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.015497 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260619 18:56:25.017570 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.017004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60380)
0619 18:56:25.017220 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.017369 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 18:56:25.018007 29246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:33667: Network error: Client connection negotiation failed: client connection to 127.15.91.61:33667: connect: Connection refused (error 111)
W20260619 18:56:25.030318 29253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:25.030673 29253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:25.030870 29253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:25.039067 29253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:56:25.039215 29253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:25.039335 29253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:25.039438 29253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:56:25.039527 29253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:56:25.051043 29253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46489,127.15.91.61:33667,127.15.91.60:40183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab
--principal=kudu/127.15.91.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.61:33667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.91.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:25.052579 29253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:25.054088 29253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:25.067979 29258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:25.068043 29261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:25.068022 29259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:25.069132 29253 server_base.cc:1034] running on GCE node
I20260619 18:56:25.070022 29253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:25.071789 29253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:25.073051 29253 hybrid_clock.cc:648] HybridClock initialized: now 1781895385072994 us; error 59 us; skew 500 ppm
I20260619 18:56:25.073500 29253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895385, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:25.077683 29253 init.cc:377] Logged in from keytab as kudu/127.15.91.61@KRBTEST.COM (short username kudu)
I20260619 18:56:25.079879 29253 webserver.cc:460] Webserver started at http://127.15.91.61:37405/ using document root <none> and password file <none>
I20260619 18:56:25.080780 29253 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:25.080946 29253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:25.081465 29253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:25.085315 29253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/data/instance:
uuid: "46665d89cd7849b2800031dce9ecaa80"
format_stamp: "Formatted at 2026-06-19 18:56:25 on dist-test-slave-dm3f"
server_key: "04a853d9b9f1aefafd69d2a6fb4d5ec8"
server_key_iv: "a6e8fb635130467f0aeb9ae76fe60241"
server_key_version: "encryptionkey@0"
I20260619 18:56:25.086333 29253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/wal/instance:
uuid: "46665d89cd7849b2800031dce9ecaa80"
format_stamp: "Formatted at 2026-06-19 18:56:25 on dist-test-slave-dm3f"
server_key: "04a853d9b9f1aefafd69d2a6fb4d5ec8"
server_key_iv: "a6e8fb635130467f0aeb9ae76fe60241"
server_key_version: "encryptionkey@0"
I20260619 18:56:25.092656 29253 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260619 18:56:25.097918 29268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:25.100243 29253 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260619 18:56:25.100540 29253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/wal
uuid: "46665d89cd7849b2800031dce9ecaa80"
format_stamp: "Formatted at 2026-06-19 18:56:25 on dist-test-slave-dm3f"
server_key: "04a853d9b9f1aefafd69d2a6fb4d5ec8"
server_key_iv: "a6e8fb635130467f0aeb9ae76fe60241"
server_key_version: "encryptionkey@0"
I20260619 18:56:25.100795 29253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:25.145599 29253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:25.152119 29253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:25.152406 29253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:25.175060 29253 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.61:33667
I20260619 18:56:25.175084 29319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.61:33667 every 8 connection(s)
I20260619 18:56:25.176744 29253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
I20260619 18:56:25.181008 29320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:25.187014 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 29253
I20260619 18:56:25.187260 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-1/wal/instance
I20260619 18:56:25.187842 15724 external_mini_cluster.cc:1428] Setting key 2e8279f393db84d0d743f88cd16774e2
I20260619 18:56:25.189823 29320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895385, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.60@KRBTEST.COM" created.
I20260619 18:56:25.215513 29322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.193949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.15.91.62:46489)
0619 18:56:25.194552 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.194560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.195922 (+ 1362us) server_negotiation.cc:407] Connection header received
0619 18:56:25.198143 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.198149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.198263 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.198397 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.200331 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.201116 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.202984 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.203451 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.203637 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.208099 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.208105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.208109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.208153 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.211726 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.212753 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.212760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.212765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.212919 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.213439 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.213450 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.213452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.213773 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.213947 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.214823 (+ 876us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.215090 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":303,"threads_started":1}
I20260619 18:56:25.216583 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.194615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46489 (local address 127.0.0.1:52650)
0619 18:56:25.195667 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.195751 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:25.197755 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:25.198707 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:25.198775 (+ 68us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:25.199393 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:25.200065 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.200087 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.201341 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.201345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.202779 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.202807 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.203771 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.203781 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.204057 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.204941 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:25.204968 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:25.207748 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:25.211969 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.211988 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.212006 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.212566 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.213057 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.213063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.213067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.213292 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.213923 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:25.213932 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:25.214378 (+ 446us) client_negotiation.cc:769] Sending connection context
0619 18:56:25.214691 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 18:56:25.215320 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":370,"threads_started":1}
I20260619 18:56:25.218538 29320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895385, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:56:25.237828 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.219507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60388)
0619 18:56:25.219755 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.219778 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:25.219914 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:25.221240 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:25.221243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:25.221649 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:25.221810 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.221819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.224159 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.224163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.225065 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.225080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.226535 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.226539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.226799 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.227476 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:25.227497 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:25.229965 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:25.234989 (+ 5024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.234994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.234997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.235503 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.235951 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.235955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.235957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.236153 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.236984 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:25.236987 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:25.237114 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 18:56:25.237245 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 18:56:25.237538 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":1}
I20260619 18:56:25.238298 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.219736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.15.91.61:33667)
0619 18:56:25.220597 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.220608 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.220650 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:56:25.220727 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.220738 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.221022 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.221229 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.222000 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.223944 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.225262 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.226347 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.226524 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.230298 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.230323 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.230350 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.230407 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.234665 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.235666 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.235674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.235676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.235816 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.236306 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.236310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.236312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.236826 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.237014 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.237466 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.237745 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":633,"thread_start_us":457,"threads_started":1}
I20260619 18:56:25.241144 29320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab.
Entry for principal kudu/127.15.91.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab.
I20260619 18:56:25.242669 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.242042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55872)
0619 18:56:25.242271 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.242427 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 18:56:25.242959 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.60:40183
--webserver_interface=127.15.91.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab
--principal=kudu/127.15.91.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46489,127.15.91.61:33667,127.15.91.60:40183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:25.243068 29269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
I20260619 18:56:25.245136 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.244563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55876)
0619 18:56:25.244809 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.244938 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 18:56:25.245456 29320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:40183: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
I20260619 18:56:25.294916 29320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } attempt: 1
I20260619 18:56:25.296150 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.295652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55884)
0619 18:56:25.295867 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.295981 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 18:56:25.297299 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.296889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55896)
0619 18:56:25.297089 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.297173 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 18:56:25.297537 29320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:40183: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
I20260619 18:56:25.317399 29246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } attempt: 5
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:56:25.332540 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.318322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.15.91.61:33667)
0619 18:56:25.318515 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.318523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.318556 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:56:25.318848 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.318856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.318949 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.319065 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.319947 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.320664 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.321908 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.322398 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.322549 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.326319 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.326325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.326329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.326370 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.329410 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.330267 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.330275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.330284 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.330427 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.330940 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.330947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.330954 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.331285 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.331459 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.331948 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.332203 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 18:56:25.332540 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.318230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60404)
0619 18:56:25.318465 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.318490 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:25.318658 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:25.319102 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:25.319108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:25.319579 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:25.319765 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.319775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.320822 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.320830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.321739 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.321751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.322576 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.322582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.322834 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.323537 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:25.323557 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:25.326022 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:25.329633 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.329640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.329644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.330099 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.330565 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.330571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.330575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.330807 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.331423 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:25.331428 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:25.331622 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 18:56:25.331808 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 18:56:25.332154 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 18:56:25.333695 29246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:56:25.335039 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.334474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55904)
0619 18:56:25.334696 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.334833 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 18:56:25.336275 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.335827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55906)
0619 18:56:25.335998 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.336125 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:56:25.336601 29246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:40183: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
I20260619 18:56:25.365787 29320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } attempt: 2
I20260619 18:56:25.367012 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.366496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55908)
0619 18:56:25.366747 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.366846 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 18:56:25.368219 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.367801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55910)
0619 18:56:25.367996 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.368080 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 18:56:25.368476 29320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:40183: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
I20260619 18:56:25.391849 29246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } attempt: 1
I20260619 18:56:25.393265 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.392649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55914)
0619 18:56:25.392937 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.393060 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260619 18:56:25.394469 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.394030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55926)
0619 18:56:25.394215 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.394314 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 18:56:25.394795 29246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:40183: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
I20260619 18:56:25.459723 29320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } attempt: 3
I20260619 18:56:25.460906 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.460422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55930)
0619 18:56:25.460632 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.460732 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 18:56:25.462291 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.461774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55932)
0619 18:56:25.462053 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.462151 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260619 18:56:25.462612 29320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:40183: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
I20260619 18:56:25.463020 29246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } attempt: 2
I20260619 18:56:25.464286 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.463594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55942)
0619 18:56:25.463875 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.464016 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 18:56:25.465698 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.465149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55954)
0619 18:56:25.465363 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.465482 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 18:56:25.466006 29246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:40183: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
I20260619 18:56:25.572328 29246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } attempt: 3
I20260619 18:56:25.573710 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.573068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55970)
0619 18:56:25.573350 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.573482 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 18:56:25.575062 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.574503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55978)
0619 18:56:25.574779 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.574889 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260619 18:56:25.575326 29246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:40183: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
I20260619 18:56:25.605866 29320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } attempt: 4
I20260619 18:56:25.607148 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.606639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:55994)
0619 18:56:25.606860 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.606959 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 18:56:25.608394 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.607988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:56010)
0619 18:56:25.608161 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.608247 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 18:56:25.608851 29320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:40183: Network error: Client connection negotiation failed: client connection to 127.15.91.60:40183: connect: Connection refused (error 111)
W20260619 18:56:25.620141 29328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:25.620478 29328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:25.620635 29328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:25.628703 29328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:56:25.628846 29328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:25.628943 29328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:25.629053 29328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:56:25.629137 29328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:56:25.640352 29328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46489,127.15.91.61:33667,127.15.91.60:40183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab
--principal=kudu/127.15.91.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.60:40183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.91.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:25.641918 29328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:25.643488 29328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:25.657378 29333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:25.657613 29336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:25.657543 29334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:25.658689 29328 server_base.cc:1034] running on GCE node
I20260619 18:56:25.659958 29328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:25.661777 29328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:25.663065 29328 hybrid_clock.cc:648] HybridClock initialized: now 1781895385663022 us; error 68 us; skew 500 ppm
I20260619 18:56:25.663515 29328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895385, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:25.667140 29328 init.cc:377] Logged in from keytab as kudu/127.15.91.60@KRBTEST.COM (short username kudu)
I20260619 18:56:25.669064 29328 webserver.cc:460] Webserver started at http://127.15.91.60:39887/ using document root <none> and password file <none>
I20260619 18:56:25.669952 29328 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:25.670145 29328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:25.670501 29328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:25.674096 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/data/instance:
uuid: "88b475ce9d3e4117bfe5d0efbf388cc5"
format_stamp: "Formatted at 2026-06-19 18:56:25 on dist-test-slave-dm3f"
server_key: "4c57005f3b4c87c1f3865f59d76f40d0"
server_key_iv: "48b1052c097d52dc0d9ae06f3ca4a327"
server_key_version: "encryptionkey@0"
I20260619 18:56:25.675204 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/wal/instance:
uuid: "88b475ce9d3e4117bfe5d0efbf388cc5"
format_stamp: "Formatted at 2026-06-19 18:56:25 on dist-test-slave-dm3f"
server_key: "4c57005f3b4c87c1f3865f59d76f40d0"
server_key_iv: "48b1052c097d52dc0d9ae06f3ca4a327"
server_key_version: "encryptionkey@0"
I20260619 18:56:25.681713 29328 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:56:25.686162 29343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:25.688015 29328 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:56:25.688304 29328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/wal
uuid: "88b475ce9d3e4117bfe5d0efbf388cc5"
format_stamp: "Formatted at 2026-06-19 18:56:25 on dist-test-slave-dm3f"
server_key: "4c57005f3b4c87c1f3865f59d76f40d0"
server_key_iv: "48b1052c097d52dc0d9ae06f3ca4a327"
server_key_version: "encryptionkey@0"
I20260619 18:56:25.688565 29328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:25.707180 29328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:25.712256 29328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:25.712626 29328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:25.734853 29328 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.60:40183
I20260619 18:56:25.734854 29394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.60:40183 every 8 connection(s)
I20260619 18:56:25.736505 29328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
I20260619 18:56:25.739472 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 29328
I20260619 18:56:25.739696 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781895164554191-15724-0/minicluster-data/master-2/wal/instance
I20260619 18:56:25.740207 15724 external_mini_cluster.cc:1428] Setting key 667d2a751166adebd9ac7573fd456afa
I20260619 18:56:25.741241 29395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:25.752650 29246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } attempt: 4
I20260619 18:56:25.752619 29395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895385, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:25.772737 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.753371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:56032)
0619 18:56:25.753668 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.753691 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:25.753820 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:25.754985 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:25.754991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:25.755573 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:25.755820 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.755836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.757746 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.757754 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.759378 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.759407 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.760696 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.760706 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.760955 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.761752 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:25.761774 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:25.764571 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:25.769309 (+ 4738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.769315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.769319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.769947 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.770472 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.770479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.770484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.770681 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.771568 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:25.771575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:25.771787 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 18:56:25.771947 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 18:56:25.772301 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260619 18:56:25.773929 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.742916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.15.91.60:40183)
0619 18:56:25.743872 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.743890 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.743949 (+ 59us) server_negotiation.cc:407] Connection header received
0619 18:56:25.746662 (+ 2713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.746706 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.747458 (+ 752us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.748024 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.749544 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.751719 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.754088 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.755235 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.755464 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.761923 (+ 6459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.761941 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.761976 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.762032 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.766865 (+ 4833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.768025 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.768037 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.768049 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.768355 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.768938 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.768947 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.768951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.769898 (+ 947us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.770256 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.771830 (+ 1574us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.772396 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":415,"threads_started":1}
I20260619 18:56:25.773967 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.753621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.15.91.60:40183)
0619 18:56:25.754509 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.754515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.754541 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:56:25.754602 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.754607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.754768 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.754902 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.756044 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.757483 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.759682 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.760436 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.761344 (+ 908us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.764856 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.764863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.764866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.764908 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.769155 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.770134 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.770141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.770145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.770331 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.770830 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.770835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.770838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.771234 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.771401 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.772126 (+ 725us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.772613 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":176,"server-negotiator.queue_time_us":718,"thread_start_us":528,"threads_started":1}
I20260619 18:56:25.783200 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.756463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46489 (local address 127.0.0.1:52664)
0619 18:56:25.762402 (+ 5939us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.762459 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:25.762652 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:25.763510 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:25.763524 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:25.764492 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:25.764988 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.765009 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.766607 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.766616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.768546 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.768564 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.769657 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.769668 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.769913 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.770663 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:25.770685 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:25.774463 (+ 3778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:25.779378 (+ 4915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.779399 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.779410 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.780136 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.780708 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.780715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.780720 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.780920 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.781775 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:25.781784 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:25.782026 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 18:56:25.782263 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 18:56:25.782679 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5504,"thread_start_us":5326,"threads_started":1}
I20260619 18:56:25.783221 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.756500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.15.91.62:46489)
0619 18:56:25.761738 (+ 5238us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.761743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.762783 (+ 1040us) server_negotiation.cc:407] Connection header received
0619 18:56:25.762968 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.762976 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.763111 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.763253 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.765196 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.766373 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.768716 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.769402 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.769579 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.774758 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.774767 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.774772 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.774824 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.779168 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.780331 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.780338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.780341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.780546 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.781087 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.781092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.781096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.781592 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.781802 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.782416 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.782694 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5085,"thread_start_us":5080,"threads_started":1}
I20260619 18:56:25.783295 29246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1: Bootstrap starting.
I20260619 18:56:25.785645 29395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:56:25.787914 29246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:25.789345 29246 log.cc:824] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:25.794566 29246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1: No bootstrap required, opened a new log
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895385, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:56:25.800658 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.777743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.15.91.62:46489)
0619 18:56:25.779302 (+ 1559us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.779307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.779336 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:56:25.779393 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.779397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.779473 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.779619 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.781516 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.782759 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.784691 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.785440 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.785611 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.791599 (+ 5988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.791605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.791608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.791647 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.796002 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.797535 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.797541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.797545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.798086 (+ 541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.798644 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.798650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.798653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.799172 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.799471 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.799910 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.800211 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1379,"thread_start_us":1148,"threads_started":1}
I20260619 18:56:25.802562 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.787033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.15.91.61:33667)
0619 18:56:25.787262 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.787269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.787291 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:56:25.787360 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.787365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.787450 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.787586 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.788648 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.789714 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.791400 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.792123 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.792242 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.796182 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.796188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.796191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.796232 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.799067 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.800113 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.800122 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.800127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.800325 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.800933 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.800939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.800942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.801284 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.801400 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.801915 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.802176 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:56:25.802556 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.786629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:33667 (local address 127.0.0.1:60406)
0619 18:56:25.786927 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.786949 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:25.787072 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:25.787663 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:25.787671 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:25.788197 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:25.788436 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.788453 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.789902 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.789912 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.791164 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.791188 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.792563 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.792569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.792744 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.793374 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:25.793405 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:25.795892 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:25.799246 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.799253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.799257 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.799938 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.800514 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.800520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.800524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.800773 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.801477 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:25.801480 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:25.801634 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 18:56:25.801794 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 18:56:25.802135 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 18:56:25.804734 29395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:56:25.808089 29246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } }
I20260619 18:56:25.808645 29246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:25.808774 29246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82fc33347e1749d8acbc2438869742c1, State: Initialized, Role: FOLLOWER
I20260619 18:56:25.809659 29246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } }
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895385, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:56:25.816125 29405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } } }
I20260619 18:56:25.816671 29405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:56:25.817866 29246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:56:25.823566 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.805697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:56040)
0619 18:56:25.806060 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.806082 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:25.806209 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:25.807244 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:25.807249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:25.807988 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:25.808205 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.808218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.809605 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.809612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.811001 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.811019 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.812162 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.812168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.812383 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.813149 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:25.813179 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:25.816681 (+ 3502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:25.820773 (+ 4092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.820780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.820785 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.821219 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.821639 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.821643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.821647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.821790 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.822383 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:25.822388 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:25.822555 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 18:56:25.822724 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 18:56:25.823146 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260619 18:56:25.823624 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.805862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.15.91.60:40183)
0619 18:56:25.806069 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.806075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.806354 (+ 279us) server_negotiation.cc:407] Connection header received
0619 18:56:25.806404 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.806412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.806788 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.807001 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.808428 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.809439 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.811180 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.811780 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.811936 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.816960 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.816967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.816973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.817022 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.820597 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.821363 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.821369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.821373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.821514 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.821900 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.821907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.821911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.822260 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.822427 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.822859 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.823303 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":195,"server-negotiator.queue_time_us":70}
W20260619 18:56:25.825214 29200 tablet.cc:2369] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:56:25.827962 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.802969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.15.91.61:33667)
0619 18:56:25.803294 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.803299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.803320 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:56:25.803384 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.803393 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.803492 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.803601 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.805726 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.807288 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.809398 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.810121 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.810313 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.817448 (+ 7135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.817456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.817461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.817509 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.822600 (+ 5091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.824111 (+ 1511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.824121 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.824127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.824373 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.825042 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.825050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.825057 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.825579 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.827052 (+ 1473us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.827233 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.827522 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":527,"threads_started":1}
I20260619 18:56:25.829981 29395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5: Bootstrap starting.
I20260619 18:56:25.833877 29395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:25.835464 29395 log.cc:824] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:25.836015 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:56:25.839385 29395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5: No bootstrap required, opened a new log
W20260619 18:56:25.853425 29417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:56:25.853785 29417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:56:25.857460 29395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } }
I20260619 18:56:25.858183 29395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:25.858306 29395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b475ce9d3e4117bfe5d0efbf388cc5, State: Initialized, Role: FOLLOWER
I20260619 18:56:25.859637 29395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } }
I20260619 18:56:25.846755 29179 master_runner.cc:186] Error getting master registration for 127.15.91.61:33667: OK, instance_id { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" instance_seqno: 1781895385168879 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:56:25.861182 29179 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:56:25.861917 29422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } } }
I20260619 18:56:25.862919 29422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:56:25.864238 29395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:56:25.908209 29320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } attempt: 5
I20260619 18:56:25.916200 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.877017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.15.91.60:40183)
0619 18:56:25.877386 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.877396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.894097 (+ 16701us) server_negotiation.cc:407] Connection header received
0619 18:56:25.894361 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.894368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.894469 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.894598 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.897148 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.898363 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.900677 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.901453 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.901580 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.906075 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.906086 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.906091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.906154 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.911241 (+ 5087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.912412 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.912418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.912422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.912645 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.913207 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.913214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.913217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.913663 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.915322 (+ 1659us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.915505 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.915793 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 18:56:25 dist-test-slave-dm3f krb5kdc[29166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895385, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
I20260619 18:56:25.931699 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.870192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.15.91.61:33667)
0619 18:56:25.870442 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.870451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.891375 (+ 20924us) server_negotiation.cc:407] Connection header received
0619 18:56:25.891650 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.891660 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.891794 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.891955 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.895225 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.896048 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.899147 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.900192 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.900364 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.920316 (+ 19952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.920324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.920329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.920382 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.927063 (+ 6681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.928176 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.928185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.928189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.928388 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.928917 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.928924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.928927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.929397 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.929670 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.930948 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.931269 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260619 18:56:25.935664 29437 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:56:25.935882 29437 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:56:25.947793 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.869621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.15.91.62:46489)
0619 18:56:25.869991 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.870014 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.888791 (+ 18777us) server_negotiation.cc:407] Connection header received
0619 18:56:25.889301 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.889310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.889439 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.889545 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.894060 (+ 4515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.895260 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.904275 (+ 9015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.905389 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.905587 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.925616 (+ 20029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.925623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.925629 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.925680 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.938490 (+ 12810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.942306 (+ 3816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.942314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.942319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.942522 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.943231 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.943236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.943239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.943688 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.945639 (+ 1951us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.947071 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.947384 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260619 18:56:25.950387 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.909367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:40183 (local address 127.0.0.1:56066)
0619 18:56:25.909900 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:25.909928 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:25.910130 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:25.911451 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:25.911459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:25.912196 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:25.912506 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.912525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.914670 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.914679 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.919411 (+ 4732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:25.919444 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.920851 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.920860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:25.921176 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.921705 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:25.921737 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:25.939462 (+ 17725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:25.946373 (+ 6911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.946380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.946385 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.947077 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.947664 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:25.947669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:25.947673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:25.947879 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:25.949232 (+ 1353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:25.949238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:25.949428 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 18:56:25.949585 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 18:56:25.949942 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346}
I20260619 18:56:25.926805 29328 master_runner.cc:186] Error getting master registration for 127.15.91.61:33667: OK, instance_id { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" instance_seqno: 1781895385168879 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:56:25.950950 29328 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:56:25.954284 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:25.910425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.15.91.60:40183)
0619 18:56:25.910743 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:25.910754 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:25.910784 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:56:25.910867 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:25.910875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:25.910983 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:25.911133 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:25.912793 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.914411 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.919667 (+ 5256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:25.920480 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:25.920627 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:25.939785 (+ 19158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:25.939795 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:25.939806 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:25.939853 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:25.946154 (+ 6301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.947282 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.947288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.947291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.947492 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:25.948058 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:25.948063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:25.948067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:25.949054 (+ 987us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:25.951672 (+ 2618us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:25.951883 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 18:56:25.952204 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:56:25.963200 29320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80: Bootstrap starting.
W20260619 18:56:25.965185 29349 tablet.cc:2369] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:56:25.967939 29320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:25.969532 29320 log.cc:824] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:25.975656 29320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80: No bootstrap required, opened a new log
I20260619 18:56:25.988240 29320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } }
I20260619 18:56:25.988802 29320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:25.988950 29320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46665d89cd7849b2800031dce9ecaa80, State: Initialized, Role: FOLLOWER
I20260619 18:56:25.989876 29320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } }
I20260619 18:56:25.994040 29440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } } }
I20260619 18:56:25.994737 29440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:56:25.995604 29320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:56:26.024228 29253 master_runner.cc:263] No leader master found from master 46665d89cd7849b2800031dce9ecaa80
I20260619 18:56:26.024534 29253 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 18:56:26.027877 29451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:56:26.028244 29451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 18:56:26.156350 29274 tablet.cc:2369] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:56:26.170570 29422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:56:26.171005 29422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } }
I20260619 18:56:26.172916 29422 leader_election.cc:288] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82fc33347e1749d8acbc2438869742c1 (127.15.91.62:46489), 46665d89cd7849b2800031dce9ecaa80 (127.15.91.61:33667)
I20260619 18:56:26.174305 29221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82fc33347e1749d8acbc2438869742c1" is_pre_election: true
I20260619 18:56:26.175252 29221 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b475ce9d3e4117bfe5d0efbf388cc5 in term 0.
I20260619 18:56:26.176198 29346 leader_election.cc:302] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82fc33347e1749d8acbc2438869742c1, 88b475ce9d3e4117bfe5d0efbf388cc5; no voters:
I20260619 18:56:26.176863 29422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:56:26.177063 29422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:56:26.177210 29422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:26.180918 29422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } }
I20260619 18:56:26.182895 29295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46665d89cd7849b2800031dce9ecaa80" is_pre_election: true
I20260619 18:56:26.183655 29295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b475ce9d3e4117bfe5d0efbf388cc5 in term 0.
I20260619 18:56:26.188656 29422 leader_election.cc:288] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [CANDIDATE]: Term 1 election: Requested vote from peers 82fc33347e1749d8acbc2438869742c1 (127.15.91.62:46489), 46665d89cd7849b2800031dce9ecaa80 (127.15.91.61:33667)
I20260619 18:56:26.189136 29221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82fc33347e1749d8acbc2438869742c1"
I20260619 18:56:26.189427 29221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:26.194303 29295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46665d89cd7849b2800031dce9ecaa80"
I20260619 18:56:26.194605 29295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:26.195497 29221 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b475ce9d3e4117bfe5d0efbf388cc5 in term 1.
I20260619 18:56:26.196213 29346 leader_election.cc:302] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82fc33347e1749d8acbc2438869742c1, 88b475ce9d3e4117bfe5d0efbf388cc5; no voters:
I20260619 18:56:26.196918 29422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:56:26.200861 29295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b475ce9d3e4117bfe5d0efbf388cc5 in term 1.
I20260619 18:56:26.218297 29422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [term 1 LEADER]: Becoming Leader. State: Replica: 88b475ce9d3e4117bfe5d0efbf388cc5, State: Running, Role: LEADER
I20260619 18:56:26.219992 29422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } }
I20260619 18:56:26.233289 29453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88b475ce9d3e4117bfe5d0efbf388cc5. Latest consensus state: current_term: 1 leader_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } } }
I20260619 18:56:26.233850 29453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:26.239529 29455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:56:26.242321 29455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:56:26.265038 29295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [term 1 FOLLOWER]: Refusing update from remote peer 88b475ce9d3e4117bfe5d0efbf388cc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:56:26.266434 29422 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:56:26.274684 29221 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [term 1 FOLLOWER]: Refusing update from remote peer 88b475ce9d3e4117bfe5d0efbf388cc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:56:26.276070 29422 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:56:26.302666 29405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88b475ce9d3e4117bfe5d0efbf388cc5. Latest consensus state: current_term: 1 leader_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } } }
I20260619 18:56:26.303099 29405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:56:26.308889 29456 mvcc.cc:205] Tried to move back new op lower bound from 7298643502127435776 to 7298643501971210240. Current Snapshot: MvccSnapshot[applied={T|T < 7298643502127435776}]
I20260619 18:56:26.315203 29405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } } }
I20260619 18:56:26.315532 29405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:56:26.319305 29422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } } }
I20260619 18:56:26.319870 29422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:26.324352 29440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88b475ce9d3e4117bfe5d0efbf388cc5. Latest consensus state: current_term: 1 leader_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } } }
I20260619 18:56:26.324769 29440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:56:26.325547 29455 catalog_manager.cc:1347] Generated new cluster ID: 1a0fe8e5a4404d10aa6ac1ae4893c22c
I20260619 18:56:26.325680 29455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:56:26.334527 29422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } } }
I20260619 18:56:26.334906 29422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:26.337169 29457 mvcc.cc:205] Tried to move back new op lower bound from 7298643502127435776 to 7298643501971210240. Current Snapshot: MvccSnapshot[applied={T|T < 7298643502127435776}]
I20260619 18:56:26.347463 29440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fc33347e1749d8acbc2438869742c1" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46489 } } peers { permanent_uuid: "46665d89cd7849b2800031dce9ecaa80" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 33667 } } peers { permanent_uuid: "88b475ce9d3e4117bfe5d0efbf388cc5" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 40183 } } }
I20260619 18:56:26.347806 29440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:56:26.361820 29455 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:56:26.364527 29455 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:56:26.389940 29455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88b475ce9d3e4117bfe5d0efbf388cc5: Generated new TSK 0
I20260619 18:56:26.391274 29455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:56:26.855980 29417 catalog_manager.cc:1259] Loaded cluster ID: 1a0fe8e5a4404d10aa6ac1ae4893c22c
I20260619 18:56:26.856106 29417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1: loading cluster ID for follower catalog manager: success
I20260619 18:56:26.859016 29417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1: acquiring CA information for follower catalog manager: success
I20260619 18:56:26.861231 29417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 82fc33347e1749d8acbc2438869742c1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:56:27.031025 29451 catalog_manager.cc:1259] Loaded cluster ID: 1a0fe8e5a4404d10aa6ac1ae4893c22c
I20260619 18:56:27.031134 29451 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80: loading cluster ID for follower catalog manager: success
I20260619 18:56:27.034430 29451 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80: acquiring CA information for follower catalog manager: success
I20260619 18:56:27.036921 29451 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46665d89cd7849b2800031dce9ecaa80: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:56:27.552708 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 29179
I20260619 18:56:27.554328 29241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:27.668023 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 29179
I20260619 18:56:27.690630 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 29253
I20260619 18:56:27.692034 29315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:27.803037 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 29253
I20260619 18:56:27.824709 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 29328
I20260619 18:56:27.825991 29390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:27.938472 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 29328
2026-06-19T18:56:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5207 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:56:28 dist-test-slave-dm3f krb5kdc[29475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:56:28 dist-test-slave-dm3f krb5kdc[29475](info): set up 2 sockets
Jun 19 18:56:28 dist-test-slave-dm3f krb5kdc[29475](info): commencing operation
krb5kdc: starting...
W20260619 18:56:29.047763 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:56:29 dist-test-slave-dm3f krb5kdc[29475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:56:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:56:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:56:29.302896 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46715
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:29.674350 29488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:29.674796 29488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:29.674971 29488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:29.683794 29488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:56:29.683941 29488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:29.683993 29488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:29.684046 29488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:56:29.684160 29488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:56:29.696141 29488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:29.697750 29488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:29.699400 29488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:29.714145 29493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:29.714063 29494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:29.714118 29496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:29.715178 29488 server_base.cc:1034] running on GCE node
I20260619 18:56:29.716255 29488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:29.718133 29488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:29.719440 29488 hybrid_clock.cc:648] HybridClock initialized: now 1781895389719390 us; error 84 us; skew 500 ppm
I20260619 18:56:29.719911 29488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:29 dist-test-slave-dm3f krb5kdc[29475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895389, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:29.723798 29488 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:56:29.725812 29488 webserver.cc:460] Webserver started at http://127.15.91.62:41889/ using document root <none> and password file <none>
I20260619 18:56:29.726821 29488 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:29.727052 29488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:29.727429 29488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:29.731050 29488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "6965709346404601b5b5254a1e099731"
format_stamp: "Formatted at 2026-06-19 18:56:29 on dist-test-slave-dm3f"
server_key: "373d51f15a72119c20039353721ddfa9"
server_key_iv: "a5412d061cf71a6d675ec537d2365542"
server_key_version: "encryptionkey@0"
I20260619 18:56:29.731987 29488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "6965709346404601b5b5254a1e099731"
format_stamp: "Formatted at 2026-06-19 18:56:29 on dist-test-slave-dm3f"
server_key: "373d51f15a72119c20039353721ddfa9"
server_key_iv: "a5412d061cf71a6d675ec537d2365542"
server_key_version: "encryptionkey@0"
I20260619 18:56:29.738667 29488 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260619 18:56:29.743266 29503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:29.745107 29488 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:56:29.745410 29488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "6965709346404601b5b5254a1e099731"
format_stamp: "Formatted at 2026-06-19 18:56:29 on dist-test-slave-dm3f"
server_key: "373d51f15a72119c20039353721ddfa9"
server_key_iv: "a5412d061cf71a6d675ec537d2365542"
server_key_version: "encryptionkey@0"
I20260619 18:56:29.745687 29488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:29.764163 29488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:29.769493 29488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:29.769881 29488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:29.791265 29488 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:46715
I20260619 18:56:29.791281 29554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:46715 every 8 connection(s)
I20260619 18:56:29.792915 29488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:56:29.797185 29555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:29.798971 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 29488
I20260619 18:56:29.799199 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:56:29.799875 15724 external_mini_cluster.cc:1428] Setting key 1d177bdb70583bb60a29b9795837f583
I20260619 18:56:29.813874 29555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731: Bootstrap starting.
Jun 19 18:56:29 dist-test-slave-dm3f krb5kdc[29475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:29.818051 29555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:29.819556 29555 log.cc:824] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:29.823913 29555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731: No bootstrap required, opened a new log
I20260619 18:56:29.829211 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:29.802047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.15.91.62:46715)
0619 18:56:29.803004 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:29.803029 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:29.803091 (+ 62us) server_negotiation.cc:407] Connection header received
0619 18:56:29.805343 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:29.805383 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:29.806075 (+ 692us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:29.806587 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:29.808286 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:29.810560 (+ 2274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:29.812291 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:29.813518 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:29.813767 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:29.818989 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:29.819003 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:29.819027 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:29.819100 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:29.823522 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:29.824540 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:29.824550 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:29.824561 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:29.824832 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:29.825331 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:29.825336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:29.825339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:29.826190 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:29.826561 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:29.827052 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 18:56:29.827620 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":374,"threads_started":1}
I20260619 18:56:29.834138 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:56:29.834466 29555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965709346404601b5b5254a1e099731" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46715 } }
I20260619 18:56:29.835021 29555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:29.835162 29555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6965709346404601b5b5254a1e099731, State: Initialized, Role: FOLLOWER
I20260619 18:56:29.835846 29555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965709346404601b5b5254a1e099731" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46715 } }
I20260619 18:56:29.836129 29555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:56:29.836277 29555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:56:29.836467 29555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:29.839427 29555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965709346404601b5b5254a1e099731" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46715 } }
I20260619 18:56:29.840085 29555 leader_election.cc:302] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6965709346404601b5b5254a1e099731; no voters:
I20260619 18:56:29.840770 29555 leader_election.cc:288] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:56:29.840950 29560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:56:29.841768 29560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [term 1 LEADER]: Becoming Leader. State: Replica: 6965709346404601b5b5254a1e099731, State: Running, Role: LEADER
I20260619 18:56:29.842450 29560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965709346404601b5b5254a1e099731" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46715 } }
I20260619 18:56:29.843216 29555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:56:29.846062 29562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6965709346404601b5b5254a1e099731. Latest consensus state: current_term: 1 leader_uuid: "6965709346404601b5b5254a1e099731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965709346404601b5b5254a1e099731" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46715 } } }
I20260619 18:56:29.846020 29561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6965709346404601b5b5254a1e099731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965709346404601b5b5254a1e099731" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46715 } } }
I20260619 18:56:29.846688 29562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:29.846822 29561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:29.849983 29568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:56:29.856879 29568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:56:29.867894 29568 catalog_manager.cc:1347] Generated new cluster ID: 31db442dfdd04c5888dfff7d178a07e7
I20260619 18:56:29.868201 29568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:56:29.882018 29568 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:56:29.883621 29568 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:56:29.900004 29568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6965709346404601b5b5254a1e099731: Generated new TSK 0
I20260619 18:56:29.901046 29568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:56:29.948922 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 29488
I20260619 18:56:29.950426 29550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:30.052214 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 29488
2026-06-19T18:56:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2110 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:56:30 dist-test-slave-dm3f krb5kdc[29583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:56:30 dist-test-slave-dm3f krb5kdc[29583](info): set up 2 sockets
Jun 19 18:56:30 dist-test-slave-dm3f krb5kdc[29583](info): commencing operation
krb5kdc: starting...
W20260619 18:56:31.187764 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:56:31 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:56:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:56:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:56:31.448388 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38525
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:38525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:31.828689 29596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:31.829097 29596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:31.829274 29596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:31.837982 29596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:56:31.838192 29596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:31.838310 29596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:31.838418 29596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:56:31.838496 29596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:56:31.849745 29596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:38525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:31.851313 29596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:31.852819 29596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:31.866894 29604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:31.866864 29602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:31.866844 29601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:31.867758 29596 server_base.cc:1034] running on GCE node
I20260619 18:56:31.868788 29596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:31.870524 29596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:31.871812 29596 hybrid_clock.cc:648] HybridClock initialized: now 1781895391871759 us; error 78 us; skew 500 ppm
I20260619 18:56:31.872283 29596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:31 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895391, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:31.880267 29596 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:56:31.882232 29596 webserver.cc:460] Webserver started at http://127.15.91.62:35695/ using document root <none> and password file <none>
I20260619 18:56:31.883215 29596 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:31.883409 29596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:31.883776 29596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:31.887336 29596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "8d1826f93b9c45d78e8da3ec2c9500e3"
format_stamp: "Formatted at 2026-06-19 18:56:31 on dist-test-slave-dm3f"
server_key: "94bb0c96327e1572bdeb0a4268f33582"
server_key_iv: "286f0b236c91fae149f2c1ce26f3b14a"
server_key_version: "encryptionkey@0"
I20260619 18:56:31.888300 29596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "8d1826f93b9c45d78e8da3ec2c9500e3"
format_stamp: "Formatted at 2026-06-19 18:56:31 on dist-test-slave-dm3f"
server_key: "94bb0c96327e1572bdeb0a4268f33582"
server_key_iv: "286f0b236c91fae149f2c1ce26f3b14a"
server_key_version: "encryptionkey@0"
I20260619 18:56:31.894649 29596 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:56:31.899134 29611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:31.900892 29596 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:56:31.901165 29596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "8d1826f93b9c45d78e8da3ec2c9500e3"
format_stamp: "Formatted at 2026-06-19 18:56:31 on dist-test-slave-dm3f"
server_key: "94bb0c96327e1572bdeb0a4268f33582"
server_key_iv: "286f0b236c91fae149f2c1ce26f3b14a"
server_key_version: "encryptionkey@0"
I20260619 18:56:31.901419 29596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:31.924888 29596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:31.929750 29596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:31.930104 29596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:31.951257 29596 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:38525
I20260619 18:56:31.951311 29662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:38525 every 8 connection(s)
I20260619 18:56:31.952912 29596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:56:31.956727 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 29596
I20260619 18:56:31.956960 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:56:31.957849 15724 external_mini_cluster.cc:1428] Setting key be9126bc18543f5897c1206842d91fa8
I20260619 18:56:31.957880 29663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:31.975432 29663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Bootstrap starting.
Jun 19 18:56:31 dist-test-slave-dm3f krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:31.979996 29663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:31.981643 29663 log.cc:824] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:31.986078 29663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: No bootstrap required, opened a new log
I20260619 18:56:31.987941 29666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:31.960251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.15.91.62:38525)
0619 18:56:31.961245 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:31.961259 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:31.961308 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:56:31.963490 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:31.963521 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:31.964077 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:31.964502 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:31.966608 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:31.968207 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:31.970411 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:31.971448 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:31.971665 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:31.977285 (+ 5620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:31.977304 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:31.977329 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:31.977398 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:31.981877 (+ 4479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:31.983158 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:31.983167 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:31.983181 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:31.983495 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:31.983982 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:31.983988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:31.983992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:31.984881 (+ 889us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:31.985276 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:31.985683 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 18:56:31.986237 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":456,"threads_started":1}
I20260619 18:56:31.996692 29663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:31.997169 29663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:31.997247 29663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1826f93b9c45d78e8da3ec2c9500e3, State: Initialized, Role: FOLLOWER
I20260619 18:56:31.997848 29663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:31.998217 29663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:56:31.998410 29663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:56:31.998664 29663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:32.001384 29663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:32.002133 29663 leader_election.cc:302] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d1826f93b9c45d78e8da3ec2c9500e3; no voters:
I20260619 18:56:32.002977 29663 leader_election.cc:288] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:56:32.003171 29668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:56:32.004856 29668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 1 LEADER]: Becoming Leader. State: Replica: 8d1826f93b9c45d78e8da3ec2c9500e3, State: Running, Role: LEADER
I20260619 18:56:32.005689 29668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:32.005981 29663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:56:32.009302 29670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1826f93b9c45d78e8da3ec2c9500e3. Latest consensus state: current_term: 1 leader_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } } }
I20260619 18:56:32.009279 29669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } } }
I20260619 18:56:32.009809 29669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:32.009809 29670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:32.010869 29675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:56:32.017359 29675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:56:32.031447 29675 catalog_manager.cc:1347] Generated new cluster ID: 1fd880b9a7874d11a97e805c4b0992e9
I20260619 18:56:32.031649 29675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:56:32.047420 29675 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:56:32.048924 29675 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:56:32.059731 29675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Generated new TSK 0
I20260619 18:56:32.061115 29675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:56:32.130482 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38525
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:32.507539 29689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:32.507879 29689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:32.507985 29689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:32.516268 29689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:32.516413 29689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:32.516561 29689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:56:32.528183 29689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:32.529572 29689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:32.531284 29689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:32.546090 29694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:32.546656 29695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:32.546993 29689 server_base.cc:1034] running on GCE node
W20260619 18:56:32.547089 29697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:32.547875 29689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:32.549058 29689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:32.550377 29689 hybrid_clock.cc:648] HybridClock initialized: now 1781895392550325 us; error 77 us; skew 500 ppm
I20260619 18:56:32.550891 29689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:32 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895392, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:32.554862 29689 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:56:32.556720 29689 webserver.cc:460] Webserver started at http://127.15.91.1:34387/ using document root <none> and password file <none>
I20260619 18:56:32.557595 29689 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:32.557777 29689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:32.558136 29689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:32.561646 29689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "10401e7ce78c44e78745b8f5ae4d13e2"
format_stamp: "Formatted at 2026-06-19 18:56:32 on dist-test-slave-dm3f"
server_key: "32a944f9ef6311d620f1f3c3a402a120"
server_key_iv: "a47c58e081d962aa746cee2f25732124"
server_key_version: "encryptionkey@0"
I20260619 18:56:32.562773 29689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "10401e7ce78c44e78745b8f5ae4d13e2"
format_stamp: "Formatted at 2026-06-19 18:56:32 on dist-test-slave-dm3f"
server_key: "32a944f9ef6311d620f1f3c3a402a120"
server_key_iv: "a47c58e081d962aa746cee2f25732124"
server_key_version: "encryptionkey@0"
I20260619 18:56:32.569423 29689 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:56:32.573984 29704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:32.575980 29689 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:56:32.576256 29689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "10401e7ce78c44e78745b8f5ae4d13e2"
format_stamp: "Formatted at 2026-06-19 18:56:32 on dist-test-slave-dm3f"
server_key: "32a944f9ef6311d620f1f3c3a402a120"
server_key_iv: "a47c58e081d962aa746cee2f25732124"
server_key_version: "encryptionkey@0"
I20260619 18:56:32.576541 29689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:32.603550 29689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:32.608886 29689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:32.609248 29689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:32.610383 29689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:32.612059 29689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:32.612191 29689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:32.612365 29689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:32.612444 29689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:32.650357 29689 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:42253
I20260619 18:56:32.650405 29816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:42253 every 8 connection(s)
I20260619 18:56:32.651867 29689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:56:32.656880 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 29689
I20260619 18:56:32.657126 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:56:32.657979 15724 external_mini_cluster.cc:1428] Setting key 18836ed3c5493bfc0adbd9e98e288b0a
Jun 19 18:56:32 dist-test-slave-dm3f krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895392, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:32.677101 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:32.654811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:44281 (local address 127.15.91.62:38525)
0619 18:56:32.655326 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:32.655333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:32.656537 (+ 1204us) server_negotiation.cc:407] Connection header received
0619 18:56:32.659377 (+ 2840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:32.659381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:32.659508 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:32.659645 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:32.661666 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:32.662428 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:32.664301 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:32.664739 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:32.664901 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:32.669703 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:32.669710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:32.669713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:32.669754 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:32.672841 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:32.674178 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:32.674182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:32.674184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:32.674330 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:32.675016 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:32.675020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:32.675022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:32.675340 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:32.675475 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:32.676596 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0619 18:56:32.676821 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":384,"thread_start_us":239,"threads_started":1}
I20260619 18:56:32.678994 29820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:32.655461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:44281)
0619 18:56:32.656254 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:32.656362 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:32.659020 (+ 2658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:32.660007 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:32.660018 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:32.660730 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:32.661436 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:32.661460 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:32.662638 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:32.662642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:32.664076 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:32.664095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:32.665023 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:32.665030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:32.665306 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:32.666077 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:32.666101 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:32.669343 (+ 3242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:32.673088 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:32.673104 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:32.673131 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:32.673989 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:32.674476 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:32.674485 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:32.674491 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:32.674871 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:32.675519 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:32.675532 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:32.676104 (+ 572us) client_negotiation.cc:769] Sending connection context
0619 18:56:32.676466 (+ 362us) client_negotiation.cc:240] Negotiation successful
0619 18:56:32.677220 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":306,"threads_started":1}
I20260619 18:56:32.680958 29817 heartbeater.cc:347] Connected to a master server at 127.15.91.62:38525
I20260619 18:56:32.681368 29817 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:32.682499 29817 heartbeater.cc:511] Master 127.15.91.62:38525 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:56:32.685410 29628 ts_manager.cc:194] Registered new tserver with Master: 10401e7ce78c44e78745b8f5ae4d13e2 (127.15.91.1:42253)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:56:32.687147 29628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:44281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:56:32.714459 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38525
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:33.083719 29823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:33.084038 29823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:33.084138 29823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:33.092682 29823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:33.092830 29823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:33.092952 29823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:56:33.103360 29823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:33.104738 29823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:33.106307 29823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:33.122067 29828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:33.122021 29829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:33.122067 29831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:33.123165 29823 server_base.cc:1034] running on GCE node
I20260619 18:56:33.123853 29823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:33.124899 29823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:33.126226 29823 hybrid_clock.cc:648] HybridClock initialized: now 1781895393126155 us; error 84 us; skew 500 ppm
I20260619 18:56:33.126938 29823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:33 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895393, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:33.130543 29823 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:56:33.132454 29823 webserver.cc:460] Webserver started at http://127.15.91.2:44929/ using document root <none> and password file <none>
I20260619 18:56:33.133286 29823 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:33.133462 29823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:33.133805 29823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:33.137207 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "13d5e886e7a24122a99700e940a8f212"
format_stamp: "Formatted at 2026-06-19 18:56:33 on dist-test-slave-dm3f"
server_key: "aa88148a984ec9042a36ac87b6cf945d"
server_key_iv: "8cc7401106df85cb083c5136e60b8d61"
server_key_version: "encryptionkey@0"
I20260619 18:56:33.138190 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "13d5e886e7a24122a99700e940a8f212"
format_stamp: "Formatted at 2026-06-19 18:56:33 on dist-test-slave-dm3f"
server_key: "aa88148a984ec9042a36ac87b6cf945d"
server_key_iv: "8cc7401106df85cb083c5136e60b8d61"
server_key_version: "encryptionkey@0"
I20260619 18:56:33.144518 29823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:56:33.148794 29838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:33.150538 29823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:56:33.150838 29823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "13d5e886e7a24122a99700e940a8f212"
format_stamp: "Formatted at 2026-06-19 18:56:33 on dist-test-slave-dm3f"
server_key: "aa88148a984ec9042a36ac87b6cf945d"
server_key_iv: "8cc7401106df85cb083c5136e60b8d61"
server_key_version: "encryptionkey@0"
I20260619 18:56:33.151098 29823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:33.172698 29823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:33.177912 29823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:33.178272 29823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:33.179407 29823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:33.181530 29823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:33.181670 29823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:33.181865 29823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:33.181937 29823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:33.219813 29823 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:38637
I20260619 18:56:33.219861 29950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:38637 every 8 connection(s)
I20260619 18:56:33.221289 29823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:56:33.230965 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 29823
I20260619 18:56:33.231168 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:56:33.231662 15724 external_mini_cluster.cc:1428] Setting key 80a23ea0b264e32e001c86ad9ce5be77
Jun 19 18:56:33 dist-test-slave-dm3f krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895393, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:33.247260 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:33.224309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:38553 (local address 127.15.91.62:38525)
0619 18:56:33.224852 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:33.224860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:33.226096 (+ 1236us) server_negotiation.cc:407] Connection header received
0619 18:56:33.228204 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:33.228210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:33.228295 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:33.228409 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:33.230283 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.231306 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.233414 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.233894 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.234065 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:33.238777 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:33.238786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:33.238792 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:33.238848 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:33.241986 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:33.243547 (+ 1561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:33.243556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:33.243559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:33.243860 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:33.244860 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:33.244866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:33.244869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:33.245262 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:33.245416 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:33.246615 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0619 18:56:33.246947 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260619 18:56:33.248900 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:33.224983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:38553)
0619 18:56:33.225877 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:33.225924 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:33.227838 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:33.228731 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:33.228742 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:33.229346 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:33.230029 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:33.230051 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.231525 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.231530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:33.233217 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:33.233238 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.234218 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.234229 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:33.234503 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:33.235388 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:33.235423 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:33.238334 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:33.242262 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:33.242279 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:33.242305 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:33.243212 (+ 907us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:33.244223 (+ 1011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:33.244232 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:33.244239 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:33.244658 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:33.245461 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:33.245474 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:33.246049 (+ 575us) client_negotiation.cc:769] Sending connection context
0619 18:56:33.246446 (+ 397us) client_negotiation.cc:240] Negotiation successful
0619 18:56:33.247200 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":329,"threads_started":1}
I20260619 18:56:33.250789 29951 heartbeater.cc:347] Connected to a master server at 127.15.91.62:38525
I20260619 18:56:33.251187 29951 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:33.252071 29951 heartbeater.cc:511] Master 127.15.91.62:38525 requested a full tablet report, sending...
I20260619 18:56:33.254150 29628 ts_manager.cc:194] Registered new tserver with Master: 13d5e886e7a24122a99700e940a8f212 (127.15.91.2:38637)
I20260619 18:56:33.255172 29628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:38553
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:56:33.291040 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38525
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:33.653558 29957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:33.653939 29957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:33.654158 29957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:33.662388 29957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:33.662595 29957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:33.662845 29957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:56:33.673601 29957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:33.675164 29957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:33.676700 29957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:56:33.690671 29817 heartbeater.cc:503] Master 127.15.91.62:38525 was elected leader, sending a full tablet report...
W20260619 18:56:33.693819 29962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:33.694020 29965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:33.693819 29963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:33.694854 29957 server_base.cc:1034] running on GCE node
I20260619 18:56:33.695647 29957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:33.696873 29957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:33.698153 29957 hybrid_clock.cc:648] HybridClock initialized: now 1781895393698102 us; error 47 us; skew 500 ppm
I20260619 18:56:33.698591 29957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:33 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895393, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:33.702914 29957 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:56:33.705184 29957 webserver.cc:460] Webserver started at http://127.15.91.3:35113/ using document root <none> and password file <none>
I20260619 18:56:33.706223 29957 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:33.706420 29957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:33.706815 29957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:33.710318 29957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f"
format_stamp: "Formatted at 2026-06-19 18:56:33 on dist-test-slave-dm3f"
server_key: "72eaa927a53db975a54a9168576c072a"
server_key_iv: "6a11157529a7112d86074083fc2cc280"
server_key_version: "encryptionkey@0"
I20260619 18:56:33.711441 29957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f"
format_stamp: "Formatted at 2026-06-19 18:56:33 on dist-test-slave-dm3f"
server_key: "72eaa927a53db975a54a9168576c072a"
server_key_iv: "6a11157529a7112d86074083fc2cc280"
server_key_version: "encryptionkey@0"
I20260619 18:56:33.718343 29957 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 18:56:33.722982 29972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:33.725014 29957 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:56:33.725291 29957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f"
format_stamp: "Formatted at 2026-06-19 18:56:33 on dist-test-slave-dm3f"
server_key: "72eaa927a53db975a54a9168576c072a"
server_key_iv: "6a11157529a7112d86074083fc2cc280"
server_key_version: "encryptionkey@0"
I20260619 18:56:33.725538 29957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:33.745225 29957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:33.750556 29957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:33.750988 29957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:33.752156 29957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:33.753887 29957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:33.753981 29957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:33.754163 29957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:33.754217 29957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:33.792104 29957 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:33483
I20260619 18:56:33.792138 30084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:33483 every 8 connection(s)
I20260619 18:56:33.793648 29957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:56:33.797513 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 29957
I20260619 18:56:33.797724 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:56:33.798202 15724 external_mini_cluster.cc:1428] Setting key 58c0830d8f17935f8f60bb427d462d00
Jun 19 18:56:33 dist-test-slave-dm3f krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895393, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:33.820356 30087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:33.796786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:38183 (local address 127.15.91.62:38525)
0619 18:56:33.797316 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:33.797323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:33.798782 (+ 1459us) server_negotiation.cc:407] Connection header received
0619 18:56:33.801060 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:33.801065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:33.801168 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:33.801275 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:33.803442 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.804312 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.806549 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.807150 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.807354 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:33.812021 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:33.812028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:33.812031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:33.812083 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:33.816013 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:33.817175 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:33.817183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:33.817186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:33.817482 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:33.818167 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:33.818172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:33.818175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:33.818606 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:33.818975 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:33.819805 (+ 830us) server_negotiation.cc:299] Negotiation successful
0619 18:56:33.820040 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":234,"threads_started":1}
I20260619 18:56:33.821522 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:33.797452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:38183)
0619 18:56:33.798506 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:33.798573 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:33.800662 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:33.801640 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:33.801655 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:33.802362 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:33.803165 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:33.803199 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.804567 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.804574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:33.806303 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:33.806329 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.807500 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.807509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:33.807913 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:33.808742 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:33.808780 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:33.811677 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:33.816224 (+ 4547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:33.816234 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:33.816251 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:33.816853 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:33.817667 (+ 814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:33.817671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:33.817674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:33.817936 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:33.818893 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:33.818902 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:33.819337 (+ 435us) client_negotiation.cc:769] Sending connection context
0619 18:56:33.819642 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 18:56:33.820133 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":468,"threads_started":1}
I20260619 18:56:33.823680 30085 heartbeater.cc:347] Connected to a master server at 127.15.91.62:38525
I20260619 18:56:33.824033 30085 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:33.824892 30085 heartbeater.cc:511] Master 127.15.91.62:38525 requested a full tablet report, sending...
I20260619 18:56:33.826956 29628 ts_manager.cc:194] Registered new tserver with Master: 3dbb2c3c3cb24b169b98958d6a2daf8f (127.15.91.3:33483)
I20260619 18:56:33.828235 29628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:38183
I20260619 18:56:33.829295 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:56:33.850495 30087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:33.834657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.15.91.62:38525)
0619 18:56:33.834990 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:33.834998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:33.835395 (+ 397us) server_negotiation.cc:407] Connection header received
0619 18:56:33.835556 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:33.835560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:33.835652 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:33.835783 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:33.837479 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.838293 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.840353 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.840833 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.840983 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:33.844140 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:33.844147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:33.844151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:33.844198 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:33.847271 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:33.848265 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:33.848270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:33.848273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:33.848435 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:33.848902 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:33.848907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:33.848909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:33.849306 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:33.849547 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:33.849904 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 18:56:33.850118 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260619 18:56:33.855818 29627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:56:33.863732 29627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:56:33.899169 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:33.880676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.15.91.2:38637)
0619 18:56:33.881755 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:33.881768 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:33.882736 (+ 968us) server_negotiation.cc:407] Connection header received
0619 18:56:33.883342 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:33.883357 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:33.883772 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:33.884007 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:33.887029 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.889331 (+ 2302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.894030 (+ 4699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.896439 (+ 2409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.896711 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:33.897049 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:33.898262 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0619 18:56:33.898591 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":723,"threads_started":1}
I20260619 18:56:33.900512 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:33.880465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38637 (local address 127.0.0.1:55174)
0619 18:56:33.882452 (+ 1987us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:33.882517 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:33.882999 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:33.884505 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:33.884518 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:33.884589 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:33.886601 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:33.886639 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.889730 (+ 3091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.889740 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:33.893620 (+ 3880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:33.893662 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.897382 (+ 3720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.897393 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:33.897820 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:33.897905 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 18:56:33.899509 (+ 1604us) client_negotiation.cc:240] Negotiation successful
0619 18:56:33.899918 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1632,"thread_start_us":1469,"threads_started":1}
I20260619 18:56:33.908697 29886 tablet_service.cc:1467] Processing CreateTablet for tablet a156ebc3d5aa4e5ba9c902e05ae6f93a (DEFAULT_TABLE table=test-table [id=70cc6cd9a0104ffd8cebd7011938920c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:33.910593 29886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a156ebc3d5aa4e5ba9c902e05ae6f93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:33.915128 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:33.883254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:33483 (local address 127.0.0.1:38424)
0619 18:56:33.902855 (+ 19601us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:33.902886 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:33.903091 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:33.904721 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:33.904731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:33.904781 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:33.905368 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:33.905390 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.908933 (+ 3543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.908940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:33.910942 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:33.910957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.912760 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.912768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:33.912996 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:33.913101 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 18:56:33.914502 (+ 1401us) client_negotiation.cc:240] Negotiation successful
0619 18:56:33.914689 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19358,"thread_start_us":19148,"threads_started":1}
I20260619 18:56:33.916128 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:33.884012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.15.91.3:33483)
0619 18:56:33.902168 (+ 18156us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:33.902184 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:33.903296 (+ 1112us) server_negotiation.cc:407] Connection header received
0619 18:56:33.903477 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:33.903490 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:33.903907 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:33.904237 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:33.905698 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.908709 (+ 3011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.911151 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.912475 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.912656 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:33.912921 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:33.915518 (+ 2597us) server_negotiation.cc:299] Negotiation successful
0619 18:56:33.915747 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17895,"thread_start_us":17700,"threads_started":1}
I20260619 18:56:33.917270 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:33.881538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42253 (local address 127.0.0.1:47874)
0619 18:56:33.901067 (+ 19529us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:33.901096 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:33.901294 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:33.909664 (+ 8370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:33.909670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:33.909716 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:33.910012 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:33.910031 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.912348 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.912355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:33.914448 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:33.914472 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.916447 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.916454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:33.916648 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:33.916680 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:56:33.916747 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 18:56:33.916965 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19365,"thread_start_us":25638,"threads_started":1}
I20260619 18:56:33.918349 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:33.882001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.15.91.1:42253)
0619 18:56:33.908858 (+ 26857us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:33.908868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:33.908903 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:56:33.909016 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:33.909025 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:33.909397 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:33.910256 (+ 859us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:33.910528 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.912120 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.914750 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:33.916185 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:33.917647 (+ 1462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:33.917871 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:33.917939 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 18:56:33.918089 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26631,"thread_start_us":26432,"threads_started":1}
I20260619 18:56:33.920836 30020 tablet_service.cc:1467] Processing CreateTablet for tablet a156ebc3d5aa4e5ba9c902e05ae6f93a (DEFAULT_TABLE table=test-table [id=70cc6cd9a0104ffd8cebd7011938920c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:33.922328 30020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a156ebc3d5aa4e5ba9c902e05ae6f93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:33.922894 29752 tablet_service.cc:1467] Processing CreateTablet for tablet a156ebc3d5aa4e5ba9c902e05ae6f93a (DEFAULT_TABLE table=test-table [id=70cc6cd9a0104ffd8cebd7011938920c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:33.924649 29752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a156ebc3d5aa4e5ba9c902e05ae6f93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:33.932104 30104 tablet_bootstrap.cc:492] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: Bootstrap starting.
I20260619 18:56:33.937100 30104 tablet_bootstrap.cc:654] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:33.937639 30105 tablet_bootstrap.cc:492] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: Bootstrap starting.
I20260619 18:56:33.939419 30104 log.cc:824] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:33.942288 30105 tablet_bootstrap.cc:654] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:33.942873 30106 tablet_bootstrap.cc:492] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2: Bootstrap starting.
I20260619 18:56:33.943617 30104 tablet_bootstrap.cc:492] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: No bootstrap required, opened a new log
I20260619 18:56:33.943825 30105 log.cc:824] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:33.943979 30104 ts_tablet_manager.cc:1397] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260619 18:56:33.946957 30106 tablet_bootstrap.cc:654] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:33.947669 30105 tablet_bootstrap.cc:492] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: No bootstrap required, opened a new log
I20260619 18:56:33.948005 30105 ts_tablet_manager.cc:1397] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260619 18:56:33.948468 30106 log.cc:824] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:33.952548 30106 tablet_bootstrap.cc:492] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2: No bootstrap required, opened a new log
I20260619 18:56:33.952999 30106 ts_tablet_manager.cc:1397] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260619 18:56:33.955364 30104 raft_consensus.cc:348] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } } peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } }
I20260619 18:56:33.956048 30104 raft_consensus.cc:374] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:33.956183 30104 raft_consensus.cc:729] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d5e886e7a24122a99700e940a8f212, State: Initialized, Role: FOLLOWER
I20260619 18:56:33.957161 30104 consensus_queue.cc:260] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } } peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } }
I20260619 18:56:33.960009 30105 raft_consensus.cc:348] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } } peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } }
I20260619 18:56:33.960632 30105 raft_consensus.cc:374] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:33.960779 30105 raft_consensus.cc:729] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbb2c3c3cb24b169b98958d6a2daf8f, State: Initialized, Role: FOLLOWER
I20260619 18:56:33.961673 30105 consensus_queue.cc:260] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } } peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } }
I20260619 18:56:33.966852 30106 raft_consensus.cc:348] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } } peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } }
I20260619 18:56:33.967732 30106 raft_consensus.cc:374] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:33.967927 30106 raft_consensus.cc:729] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10401e7ce78c44e78745b8f5ae4d13e2, State: Initialized, Role: FOLLOWER
I20260619 18:56:33.969157 30106 consensus_queue.cc:260] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } } peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } }
I20260619 18:56:33.971632 29951 heartbeater.cc:503] Master 127.15.91.62:38525 was elected leader, sending a full tablet report...
W20260619 18:56:33.973668 29952 tablet.cc:2369] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:56:33.974165 30104 ts_tablet_manager.cc:1428] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: Time spent starting tablet: real 0.030s user 0.023s sys 0.004s
I20260619 18:56:33.982873 30085 heartbeater.cc:503] Master 127.15.91.62:38525 was elected leader, sending a full tablet report...
I20260619 18:56:33.983920 30105 ts_tablet_manager.cc:1428] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: Time spent starting tablet: real 0.036s user 0.013s sys 0.021s
I20260619 18:56:33.985267 30106 ts_tablet_manager.cc:1428] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2: Time spent starting tablet: real 0.032s user 0.015s sys 0.013s
I20260619 18:56:34.019001 30111 raft_consensus.cc:482] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:56:34.019402 30111 raft_consensus.cc:504] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } } peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } }
I20260619 18:56:34.022372 30111 leader_election.cc:288] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d5e886e7a24122a99700e940a8f212 (127.15.91.2:38637), 10401e7ce78c44e78745b8f5ae4d13e2 (127.15.91.1:42253)
I20260619 18:56:34.029122 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:34.022110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:53295 (local address 127.15.91.2:38637)
0619 18:56:34.022401 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:34.022410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:34.022448 (+ 38us) server_negotiation.cc:407] Connection header received
0619 18:56:34.022781 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:34.022789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:34.022894 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:34.023011 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:34.023560 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.024672 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.026860 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.028141 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.028271 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:34.028355 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:34.028787 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 18:56:34.028905 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 18:56:34.029307 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:34.021897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38637 (local address 127.15.91.3:53295)
0619 18:56:34.022320 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:34.022351 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:34.022511 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:34.023058 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:34.023064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:34.023097 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:34.023334 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:34.023345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.024841 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.024847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:34.026598 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:34.026616 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.028353 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.028359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:34.028575 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:34.028621 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 18:56:34.028759 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 18:56:34.028931 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1}
I20260619 18:56:34.030563 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:34.022838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:41465 (local address 127.15.91.1:42253)
0619 18:56:34.023337 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:34.023345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:34.023457 (+ 112us) server_negotiation.cc:407] Connection header received
0619 18:56:34.023688 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:34.023692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:34.023760 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:34.023838 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:34.024497 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.025607 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.027972 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.029242 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.029426 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:34.029560 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:34.030172 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 18:56:34.030307 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
I20260619 18:56:34.030767 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:34.022039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42253 (local address 127.15.91.3:41465)
0619 18:56:34.023334 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:34.023355 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:34.023530 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:34.023903 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:34.023909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:34.023939 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:34.024315 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:34.024328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.025825 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.025829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:34.027734 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:34.027757 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.029733 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.029789 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:34.029949 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:34.029977 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:56:34.030118 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 18:56:34.030294 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":1,"thread_start_us":857,"threads_started":1}
I20260619 18:56:34.030651 29906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a156ebc3d5aa4e5ba9c902e05ae6f93a" candidate_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d5e886e7a24122a99700e940a8f212" is_pre_election: true
I20260619 18:56:34.031464 29906 raft_consensus.cc:2454] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dbb2c3c3cb24b169b98958d6a2daf8f in term 0.
I20260619 18:56:34.032312 29974 leader_election.cc:302] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d5e886e7a24122a99700e940a8f212, 3dbb2c3c3cb24b169b98958d6a2daf8f; no voters:
I20260619 18:56:34.032174 29772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a156ebc3d5aa4e5ba9c902e05ae6f93a" candidate_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" is_pre_election: true
I20260619 18:56:34.032838 30111 raft_consensus.cc:2781] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:56:34.032871 29772 raft_consensus.cc:2454] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dbb2c3c3cb24b169b98958d6a2daf8f in term 0.
I20260619 18:56:34.033020 30111 raft_consensus.cc:482] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:56:34.033143 30111 raft_consensus.cc:3037] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:34.036010 30111 raft_consensus.cc:504] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } } peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } }
I20260619 18:56:34.036748 30111 leader_election.cc:288] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [CANDIDATE]: Term 1 election: Requested vote from peers 13d5e886e7a24122a99700e940a8f212 (127.15.91.2:38637), 10401e7ce78c44e78745b8f5ae4d13e2 (127.15.91.1:42253)
I20260619 18:56:34.037508 29772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a156ebc3d5aa4e5ba9c902e05ae6f93a" candidate_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10401e7ce78c44e78745b8f5ae4d13e2"
I20260619 18:56:34.037503 29906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a156ebc3d5aa4e5ba9c902e05ae6f93a" candidate_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d5e886e7a24122a99700e940a8f212"
I20260619 18:56:34.037782 29772 raft_consensus.cc:3037] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:34.037782 29906 raft_consensus.cc:3037] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:34.041450 29772 raft_consensus.cc:2454] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dbb2c3c3cb24b169b98958d6a2daf8f in term 1.
I20260619 18:56:34.041450 29906 raft_consensus.cc:2454] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dbb2c3c3cb24b169b98958d6a2daf8f in term 1.
I20260619 18:56:34.042135 29975 leader_election.cc:302] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10401e7ce78c44e78745b8f5ae4d13e2, 3dbb2c3c3cb24b169b98958d6a2daf8f; no voters:
I20260619 18:56:34.042562 30111 raft_consensus.cc:2781] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:56:34.043094 30111 raft_consensus.cc:686] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 1 LEADER]: Becoming Leader. State: Replica: 3dbb2c3c3cb24b169b98958d6a2daf8f, State: Running, Role: LEADER
I20260619 18:56:34.043644 30111 consensus_queue.cc:237] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } } peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } }
W20260619 18:56:34.045485 30086 tablet.cc:2369] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:56:34.049091 29628 catalog_manager.cc:5495] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dbb2c3c3cb24b169b98958d6a2daf8f (127.15.91.3). New cstate: current_term: 1 leader_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } health_report { overall_health: UNKNOWN } } }
I20260619 18:56:34.079990 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:34.073883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.15.91.3:33483)
0619 18:56:34.074129 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:34.074137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:34.074160 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:56:34.074420 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:34.074424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:34.074504 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:34.074686 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:56:34.075145 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.075874 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.077472 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.077913 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.078089 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:34.078658 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:56:34.079296 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:56:34.079479 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:34.079593 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 18:56:34.079689 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 18:56:34.090548 29906 raft_consensus.cc:1261] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212 [term 1 FOLLOWER]: Refusing update from remote peer 3dbb2c3c3cb24b169b98958d6a2daf8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:56:34.090548 29772 raft_consensus.cc:1261] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2 [term 1 FOLLOWER]: Refusing update from remote peer 3dbb2c3c3cb24b169b98958d6a2daf8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:56:34.091945 30111 consensus_queue.cc:1035] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:56:34.092329 30114 consensus_queue.cc:1035] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:56:34.107182 30117 mvcc.cc:205] Tried to move back new op lower bound from 7298643534181236736 to 7298643534007521280. Current Snapshot: MvccSnapshot[applied={T|T < 7298643534181236736}]
I20260619 18:56:34.123428 29627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:56:34.123770 29627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:56:34.128034 29627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Sending DeleteTablet for 3 replicas of tablet a156ebc3d5aa4e5ba9c902e05ae6f93a
I20260619 18:56:34.129601 30020 tablet_service.cc:1514] Processing DeleteTablet for tablet a156ebc3d5aa4e5ba9c902e05ae6f93a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:34 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:38424
I20260619 18:56:34.129662 29752 tablet_service.cc:1514] Processing DeleteTablet for tablet a156ebc3d5aa4e5ba9c902e05ae6f93a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:34 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:47874
I20260619 18:56:34.129624 29886 tablet_service.cc:1514] Processing DeleteTablet for tablet a156ebc3d5aa4e5ba9c902e05ae6f93a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:34 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:55174
I20260619 18:56:34.130402 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 29689
I20260619 18:56:34.140565 30126 tablet_replica.cc:330] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2: stopping tablet replica
I20260619 18:56:34.140856 30127 tablet_replica.cc:330] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: stopping tablet replica
I20260619 18:56:34.141155 30126 raft_consensus.cc:2229] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:56:34.141327 30127 raft_consensus.cc:2229] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:56:34.141788 30126 raft_consensus.cc:2258] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:34.142181 30127 raft_consensus.cc:2258] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:34.144709 30127 ts_tablet_manager.cc:1905] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:34.144701 30126 ts_tablet_manager.cc:1905] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:34.148362 30128 tablet_replica.cc:330] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: stopping tablet replica
I20260619 18:56:34.148963 30128 raft_consensus.cc:2229] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:56:34.149673 30128 raft_consensus.cc:2258] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:34.153211 30126 ts_tablet_manager.cc:1918] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:56:34.153220 30127 ts_tablet_manager.cc:1918] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:56:34.153434 30126 log.cc:1196] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/a156ebc3d5aa4e5ba9c902e05ae6f93a
I20260619 18:56:34.153663 30127 log.cc:1196] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/a156ebc3d5aa4e5ba9c902e05ae6f93a
I20260619 18:56:34.154084 30126 ts_tablet_manager.cc:1939] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 10401e7ce78c44e78745b8f5ae4d13e2: Deleting consensus metadata
I20260619 18:56:34.154425 30127 ts_tablet_manager.cc:1939] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 3dbb2c3c3cb24b169b98958d6a2daf8f: Deleting consensus metadata
I20260619 18:56:34.156809 29613 catalog_manager.cc:4841] TS 3dbb2c3c3cb24b169b98958d6a2daf8f (127.15.91.3:33483): tablet a156ebc3d5aa4e5ba9c902e05ae6f93a (table test-table [id=70cc6cd9a0104ffd8cebd7011938920c]) successfully deleted
I20260619 18:56:34.157572 30128 ts_tablet_manager.cc:1905] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:34.158740 29614 catalog_manager.cc:4841] TS 10401e7ce78c44e78745b8f5ae4d13e2 (127.15.91.1:42253): tablet a156ebc3d5aa4e5ba9c902e05ae6f93a (table test-table [id=70cc6cd9a0104ffd8cebd7011938920c]) successfully deleted
Jun 19 18:56:34 dist-test-slave-dm3f krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:56:34.164444 30128 ts_tablet_manager.cc:1918] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:56:34.164677 30128 log.cc:1196] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/a156ebc3d5aa4e5ba9c902e05ae6f93a
I20260619 18:56:34.165225 30128 ts_tablet_manager.cc:1939] T a156ebc3d5aa4e5ba9c902e05ae6f93a P 13d5e886e7a24122a99700e940a8f212: Deleting consensus metadata
I20260619 18:56:34.167174 29613 catalog_manager.cc:4841] TS 13d5e886e7a24122a99700e940a8f212 (127.15.91.2:38637): tablet a156ebc3d5aa4e5ba9c902e05ae6f93a (table test-table [id=70cc6cd9a0104ffd8cebd7011938920c]) successfully deleted
I20260619 18:56:34.170892 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:34.132042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.15.91.1:42253)
0619 18:56:34.132279 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:34.132288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:34.148029 (+ 15741us) server_negotiation.cc:407] Connection header received
0619 18:56:34.148491 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:34.148497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:34.148622 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:34.148703 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:34.151097 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.152372 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.154745 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.155491 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.155638 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:34.162284 (+ 6646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:34.162306 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:34.162328 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:34.162398 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:34.167121 (+ 4723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:34.168192 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:34.168205 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:34.168210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:34.168442 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:34.168945 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:34.168951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:34.168954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:34.169500 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:34.169884 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:34.170125 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 18:56:34.170399 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:56:34.171814 29812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:34.279670 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 29689
I20260619 18:56:34.303774 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 29823
Jun 19 18:56:34 dist-test-slave-dm3f krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:56:34.330441 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:34.304700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.15.91.2:38637)
0619 18:56:34.304909 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:34.304918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:34.304942 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:56:34.305197 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:34.305203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:34.305301 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:34.305413 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:34.307115 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.308379 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.309969 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.310670 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.310879 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:34.321065 (+ 10186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:34.321087 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:34.321110 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:34.321174 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:34.325883 (+ 4709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:34.327540 (+ 1657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:34.327555 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:34.327559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:34.327806 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:34.328339 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:34.328345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:34.328348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:34.328975 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:34.329302 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:34.329728 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 18:56:34.330024 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 18:56:34.331378 29946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:34.427763 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 29823
I20260619 18:56:34.447986 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 29957
Jun 19 18:56:34 dist-test-slave-dm3f krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:56:34.467264 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:34.449177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.15.91.3:33483)
0619 18:56:34.449410 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:34.449418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:34.449443 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:56:34.449512 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:34.449516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:34.449600 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:34.449737 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:34.451602 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.452524 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.454042 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:34.454537 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:34.454724 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:34.459783 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:34.459805 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:34.459822 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:34.459884 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:34.463502 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:34.464440 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:34.464448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:34.464451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:34.464657 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:34.465135 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:34.465146 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:34.465149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:34.465623 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:34.465988 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:34.466523 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 18:56:34.466840 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 18:56:34.468031 30080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:34.599776 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 29957
I20260619 18:56:34.621379 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 29596
I20260619 18:56:34.622673 29658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:34.737300 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 29596
I20260619 18:56:34.760933 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38525
--webserver_interface=127.15.91.62
--webserver_port=35695
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:38525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:35.135229 30134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:35.135569 30134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:35.135667 30134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:35.143925 30134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:56:35.144074 30134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:35.144124 30134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:35.144248 30134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:56:35.144322 30134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:56:35.155131 30134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:38525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=35695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:35.156663 30134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:35.158226 30134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:35.184099 30141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:35.184088 30143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:35.184088 30140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:35.192911 30134 server_base.cc:1034] running on GCE node
I20260619 18:56:35.194072 30134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:35.196170 30134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:35.197598 30134 hybrid_clock.cc:648] HybridClock initialized: now 1781895395197456 us; error 98 us; skew 500 ppm
I20260619 18:56:35.198283 30134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:35 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895395, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:35.203384 30134 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:56:35.205969 30134 webserver.cc:460] Webserver started at http://127.15.91.62:35695/ using document root <none> and password file <none>
I20260619 18:56:35.207139 30134 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:35.207368 30134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:35.216267 30134 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:56:35.220274 30150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:35.222834 30134 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260619 18:56:35.223140 30134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "8d1826f93b9c45d78e8da3ec2c9500e3"
format_stamp: "Formatted at 2026-06-19 18:56:31 on dist-test-slave-dm3f"
server_key: "94bb0c96327e1572bdeb0a4268f33582"
server_key_iv: "286f0b236c91fae149f2c1ce26f3b14a"
server_key_version: "encryptionkey@0"
I20260619 18:56:35.224483 30134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:35.249344 30134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:35.255828 30134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:35.256213 30134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:35.278618 30134 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:38525
I20260619 18:56:35.278640 30201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:38525 every 8 connection(s)
I20260619 18:56:35.280542 30134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:56:35.288821 30202 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:56:35.289877 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 30134
I20260619 18:56:35.291020 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:42253
--local_ip_for_outbound_sockets=127.15.91.1
--tserver_master_addrs=127.15.91.62:38525
--webserver_port=34387
--webserver_interface=127.15.91.1
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:56:35.294903 30202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Bootstrap starting.
I20260619 18:56:35.319434 30202 log.cc:824] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:35.335701 30202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:56:35.336346 30202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Bootstrap complete.
I20260619 18:56:35.347144 30202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:35.348146 30202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1826f93b9c45d78e8da3ec2c9500e3, State: Initialized, Role: FOLLOWER
I20260619 18:56:35.348932 30202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:35.349244 30202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:56:35.349380 30202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:56:35.349601 30202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:56:35.354435 30202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:35.355176 30202 leader_election.cc:302] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d1826f93b9c45d78e8da3ec2c9500e3; no voters:
I20260619 18:56:35.355959 30202 leader_election.cc:288] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:56:35.356115 30206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:56:35.357589 30206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 2 LEADER]: Becoming Leader. State: Replica: 8d1826f93b9c45d78e8da3ec2c9500e3, State: Running, Role: LEADER
I20260619 18:56:35.358321 30206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:35.358990 30202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:56:35.361291 30208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1826f93b9c45d78e8da3ec2c9500e3. Latest consensus state: current_term: 2 leader_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } } }
I20260619 18:56:35.361788 30208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:35.362444 30207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } } }
I20260619 18:56:35.362862 30207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:35.373103 30212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:56:35.385681 30212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:56:35.387977 30212 catalog_manager.cc:1259] Loaded cluster ID: 1fd880b9a7874d11a97e805c4b0992e9
I20260619 18:56:35.388124 30212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:56:35.393464 30212 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 18:56:35.421710 30212 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 18:56:35.427595 30223 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:56:35.433104 30212 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:56:35.435221 30212 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Loaded TSK: 0
I20260619 18:56:35.437207 30212 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:56:35.721561 30204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:35.721937 30204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:35.722119 30204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:35.731302 30204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:35.731487 30204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:35.731739 30204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:56:35.743116 30204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:42253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=34387
--tserver_master_addrs=127.15.91.62:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:35.744623 30204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:35.746192 30204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:35.763130 30229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:35.763135 30230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:35.764940 30232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:35.765038 30204 server_base.cc:1034] running on GCE node
I20260619 18:56:35.766041 30204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:35.767609 30204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:35.768947 30204 hybrid_clock.cc:648] HybridClock initialized: now 1781895395768861 us; error 85 us; skew 500 ppm
I20260619 18:56:35.769448 30204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:35 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895395, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:35.774557 30204 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:56:35.776734 30204 webserver.cc:460] Webserver started at http://127.15.91.1:34387/ using document root <none> and password file <none>
I20260619 18:56:35.777638 30204 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:35.777832 30204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:35.784893 30204 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 18:56:35.788848 30239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:35.790935 30204 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:56:35.791188 30204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "10401e7ce78c44e78745b8f5ae4d13e2"
format_stamp: "Formatted at 2026-06-19 18:56:32 on dist-test-slave-dm3f"
server_key: "32a944f9ef6311d620f1f3c3a402a120"
server_key_iv: "a47c58e081d962aa746cee2f25732124"
server_key_version: "encryptionkey@0"
I20260619 18:56:35.792484 30204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:35.821465 30204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:35.826692 30204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:35.827064 30204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:35.828262 30204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:35.830084 30204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:35.830235 30204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:35.830361 30204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:35.830430 30204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:35.872041 30204 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:42253
I20260619 18:56:35.872146 30351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:42253 every 8 connection(s)
I20260619 18:56:35.873754 30204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:56:35.875049 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 30204
I20260619 18:56:35.875962 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:38637
--local_ip_for_outbound_sockets=127.15.91.2
--tserver_master_addrs=127.15.91.62:38525
--webserver_port=44929
--webserver_interface=127.15.91.2
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:56:35 dist-test-slave-dm3f krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895395, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:35.908706 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:35.877715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:46883)
0619 18:56:35.878858 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:35.878928 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:35.882059 (+ 3131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:35.884179 (+ 2120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:35.884189 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:35.884810 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:35.885446 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:35.885468 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:35.888366 (+ 2898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:35.888372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:35.889860 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:35.889881 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:35.892046 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:35.892056 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:35.892304 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:35.893155 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:35.893184 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:35.895821 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:35.901793 (+ 5972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:35.901802 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:35.901818 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:35.902435 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:35.903269 (+ 834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:35.903273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:35.903276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:35.903521 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:35.905418 (+ 1897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:35.905426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:35.906152 (+ 726us) client_negotiation.cc:769] Sending connection context
0619 18:56:35.906572 (+ 420us) client_negotiation.cc:240] Negotiation successful
0619 18:56:35.907467 (+ 895us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":424,"threads_started":1}
I20260619 18:56:35.909003 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:35.877465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:46883 (local address 127.15.91.62:38525)
0619 18:56:35.878226 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:35.878250 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:35.879105 (+ 855us) server_negotiation.cc:407] Connection header received
0619 18:56:35.882996 (+ 3891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:35.883014 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:35.883466 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:35.883859 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:35.885738 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:35.888072 (+ 2334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:35.890118 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:35.891579 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:35.891856 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:35.896245 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:35.896262 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:35.896288 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:35.896367 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:35.901449 (+ 5082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:35.902684 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:35.902696 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:35.902708 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:35.903067 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:35.903699 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:35.903707 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:35.903713 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:35.905115 (+ 1402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:35.905404 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:35.907081 (+ 1677us) server_negotiation.cc:299] Negotiation successful
0619 18:56:35.907690 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":319,"threads_started":1}
I20260619 18:56:35.912804 30352 heartbeater.cc:347] Connected to a master server at 127.15.91.62:38525
I20260619 18:56:35.913226 30352 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:35.914291 30352 heartbeater.cc:511] Master 127.15.91.62:38525 requested a full tablet report, sending...
I20260619 18:56:35.917456 30167 ts_manager.cc:194] Registered new tserver with Master: 10401e7ce78c44e78745b8f5ae4d13e2 (127.15.91.1:42253)
I20260619 18:56:35.919224 30167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:46883
W20260619 18:56:36.265424 30356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:36.265774 30356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:36.265878 30356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:36.273808 30356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:36.273959 30356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:36.274080 30356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:56:36.285467 30356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:38637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=44929
--tserver_master_addrs=127.15.91.62:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:36.287132 30356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:36.288888 30356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:36.305377 30361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:36.306528 30362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:36.306507 30364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:36.307260 30356 server_base.cc:1034] running on GCE node
I20260619 18:56:36.308018 30356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:36.309264 30356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:36.310539 30356 hybrid_clock.cc:648] HybridClock initialized: now 1781895396310485 us; error 44 us; skew 500 ppm
I20260619 18:56:36.311049 30356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:36 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895396, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:36.315088 30356 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:56:36.317622 30356 webserver.cc:460] Webserver started at http://127.15.91.2:44929/ using document root <none> and password file <none>
I20260619 18:56:36.318493 30356 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:36.318693 30356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:36.325830 30356 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 18:56:36.330376 30371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:36.332369 30356 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:56:36.332675 30356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "13d5e886e7a24122a99700e940a8f212"
format_stamp: "Formatted at 2026-06-19 18:56:33 on dist-test-slave-dm3f"
server_key: "aa88148a984ec9042a36ac87b6cf945d"
server_key_iv: "8cc7401106df85cb083c5136e60b8d61"
server_key_version: "encryptionkey@0"
I20260619 18:56:36.333811 30356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:36.362107 30356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:36.367159 30356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:36.367486 30356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:36.368844 30356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:36.370474 30356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:36.370606 30356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:36.370812 30356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:36.370915 30356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:36.411123 30356 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:38637
I20260619 18:56:36.411188 30483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:38637 every 8 connection(s)
I20260619 18:56:36.412837 30356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:56:36.419526 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 30356
I20260619 18:56:36.420635 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:33483
--local_ip_for_outbound_sockets=127.15.91.3
--tserver_master_addrs=127.15.91.62:38525
--webserver_port=35113
--webserver_interface=127.15.91.3
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:56:36 dist-test-slave-dm3f krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895396, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:36.443506 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:36.416310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:37989 (local address 127.15.91.62:38525)
0619 18:56:36.417088 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:36.417098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:36.418018 (+ 920us) server_negotiation.cc:407] Connection header received
0619 18:56:36.422073 (+ 4055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:36.422079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:36.422193 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:36.422330 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:36.425510 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.426343 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.429437 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.430236 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.430466 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:36.435861 (+ 5395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:36.435867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:36.435871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:36.435916 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:36.439336 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:36.440324 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:36.440337 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:36.440339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:36.440499 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:36.441028 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:36.441040 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:36.441043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:36.441372 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:36.441550 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:36.442406 (+ 856us) server_negotiation.cc:299] Negotiation successful
0619 18:56:36.443073 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":411,"threads_started":1}
I20260619 18:56:36.444765 30487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:36.416666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:37989)
0619 18:56:36.417736 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:36.417801 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:36.421308 (+ 3507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:36.423015 (+ 1707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:36.423040 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:36.424103 (+ 1063us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:36.425146 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:36.425186 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.426610 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.426620 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:36.429156 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:36.429188 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.430641 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.430654 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:36.431222 (+ 568us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:36.432161 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:36.432202 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:36.435482 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:36.439551 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:36.439571 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:36.439600 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:36.440141 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:36.440639 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:36.440645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:36.440649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:36.440883 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:36.441527 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:36.441537 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:36.441962 (+ 425us) client_negotiation.cc:769] Sending connection context
0619 18:56:36.442294 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 18:56:36.443416 (+ 1122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":416,"threads_started":1}
I20260619 18:56:36.447329 30484 heartbeater.cc:347] Connected to a master server at 127.15.91.62:38525
I20260619 18:56:36.447883 30484 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:36.449041 30484 heartbeater.cc:511] Master 127.15.91.62:38525 requested a full tablet report, sending...
I20260619 18:56:36.451422 30167 ts_manager.cc:194] Registered new tserver with Master: 13d5e886e7a24122a99700e940a8f212 (127.15.91.2:38637)
I20260619 18:56:36.452737 30167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:37989
W20260619 18:56:36.804265 30488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:36.804616 30488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:36.804728 30488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:36.813067 30488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:36.813217 30488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:36.813482 30488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:56:36.824633 30488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:33483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=35113
--tserver_master_addrs=127.15.91.62:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:36.826049 30488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:36.827481 30488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:36.844583 30494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:36.844748 30493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:36.844725 30496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:36.845538 30488 server_base.cc:1034] running on GCE node
I20260619 18:56:36.846515 30488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:36.848140 30488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:36.849435 30488 hybrid_clock.cc:648] HybridClock initialized: now 1781895396849347 us; error 81 us; skew 500 ppm
I20260619 18:56:36.849855 30488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:36 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895396, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:36.853788 30488 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:56:36.855794 30488 webserver.cc:460] Webserver started at http://127.15.91.3:35113/ using document root <none> and password file <none>
I20260619 18:56:36.856669 30488 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:36.856822 30488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:36.864017 30488 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 18:56:36.867461 30503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:36.869287 30488 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260619 18:56:36.869554 30488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f"
format_stamp: "Formatted at 2026-06-19 18:56:33 on dist-test-slave-dm3f"
server_key: "72eaa927a53db975a54a9168576c072a"
server_key_iv: "6a11157529a7112d86074083fc2cc280"
server_key_version: "encryptionkey@0"
I20260619 18:56:36.870798 30488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:36.895433 30488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:36.900795 30488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:36.901124 30488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:36.902268 30488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:36.904021 30488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:36.904173 30488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:36.904333 30488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:36.904435 30488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:36.923578 30352 heartbeater.cc:503] Master 127.15.91.62:38525 was elected leader, sending a full tablet report...
I20260619 18:56:36.945096 30488 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:33483
I20260619 18:56:36.945212 30615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:33483 every 8 connection(s)
I20260619 18:56:36.946887 30488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:56:36.954144 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 30488
Jun 19 18:56:36 dist-test-slave-dm3f krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895396, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:36.972602 30620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:36.955261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.15.91.62:38525)
0619 18:56:36.955883 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:36.955887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:36.956364 (+ 477us) server_negotiation.cc:407] Connection header received
0619 18:56:36.956619 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:36.956624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:36.956715 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:36.956847 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:36.958561 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.959555 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.961126 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.961591 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.961764 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:36.965369 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:36.965376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:36.965384 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:36.965440 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:36.969079 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:36.970032 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:36.970041 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:36.970047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:36.970259 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:36.970789 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:36.970794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:36.970797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:36.971222 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:36.971493 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:36.971879 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 18:56:36.972176 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":363,"threads_started":1}
I20260619 18:56:36.973264 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:36.949791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:37621 (local address 127.15.91.62:38525)
0619 18:56:36.950316 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:36.950323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:36.951593 (+ 1270us) server_negotiation.cc:407] Connection header received
0619 18:56:36.953707 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:36.953714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:36.953827 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:36.953987 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:36.956468 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.957337 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.960111 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.960647 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.960804 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:36.965589 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:36.965595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:36.965598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:36.965642 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:36.969198 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:36.970155 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:36.970160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:36.970162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:36.970344 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:36.971021 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:36.971025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:36.971027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:36.971358 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:36.971520 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:36.972568 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0619 18:56:36.972840 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":242,"threads_started":1}
I20260619 18:56:36.974815 30619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:36.950446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:37621)
0619 18:56:36.951348 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:36.951402 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:36.953270 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:36.954242 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:36.954252 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:36.955142 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:36.956213 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:36.956248 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.957552 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.957558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:36.959726 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:36.959760 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.960984 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.960998 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:36.961285 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:36.962121 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:36.962164 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:36.965281 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:36.969403 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:36.969412 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:36.969429 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:36.970000 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:36.970505 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:36.970508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:36.970511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:36.970879 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:36.971528 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:36.971537 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:36.972047 (+ 510us) client_negotiation.cc:769] Sending connection context
0619 18:56:36.972444 (+ 397us) client_negotiation.cc:240] Negotiation successful
0619 18:56:36.973115 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":380,"threads_started":1}
I20260619 18:56:36.976828 30616 heartbeater.cc:347] Connected to a master server at 127.15.91.62:38525
I20260619 18:56:36.977221 30616 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:36.978212 30616 heartbeater.cc:511] Master 127.15.91.62:38525 requested a full tablet report, sending...
I20260619 18:56:36.980067 30167 ts_manager.cc:194] Registered new tserver with Master: 3dbb2c3c3cb24b169b98958d6a2daf8f (127.15.91.3:33483)
I20260619 18:56:36.981194 30167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:37621
I20260619 18:56:36.989982 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:56:36.998394 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:36.991269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.15.91.62:38525)
0619 18:56:36.991460 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:36.991465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:36.991784 (+ 319us) server_negotiation.cc:407] Connection header received
0619 18:56:36.992038 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:36.992046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:36.992230 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:36.992399 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:56:36.992865 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.993586 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.995416 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:36.996025 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:36.996195 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:36.997015 (+ 820us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:56:36.997742 (+ 727us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:56:36.997878 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:36.998053 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 18:56:36.998150 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:56:36.999660 30167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:56:37.007505 30167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:56:37.047168 30628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.023281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:33483 (local address 127.0.0.1:51642)
0619 18:56:37.035962 (+ 12681us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:37.036039 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:37.036316 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:37.038334 (+ 2018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:37.038347 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:37.038399 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:37.039770 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:37.039785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.042240 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.042246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:37.044262 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:37.044281 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.045987 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.045999 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:37.046248 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.046314 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 18:56:37.046493 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 18:56:37.046766 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12343,"thread_start_us":13426,"threads_started":1}
I20260619 18:56:37.047470 30629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.023968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.15.91.3:33483)
0619 18:56:37.037037 (+ 13069us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:37.037044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:37.037076 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:56:37.037220 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:37.037229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:37.037597 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:37.037842 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:37.040017 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.042060 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.044503 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.045728 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.045908 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.046193 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:37.046887 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 18:56:37.047084 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12876,"thread_start_us":12540,"threads_started":1}
I20260619 18:56:37.047582 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.024294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.15.91.1:42253)
0619 18:56:37.034828 (+ 10534us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:37.034844 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:37.037220 (+ 2376us) server_negotiation.cc:407] Connection header received
0619 18:56:37.037543 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:37.037552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:37.037847 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:37.038082 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:37.039940 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.041610 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.044707 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.046285 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.046438 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.046708 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:37.047005 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 18:56:37.047223 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10214,"thread_start_us":10561,"threads_started":1}
I20260619 18:56:37.048628 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.023463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42253 (local address 127.0.0.1:33224)
0619 18:56:37.037052 (+ 13589us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:37.037075 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:37.037244 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:37.038756 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:37.038764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:37.038810 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:37.039653 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:37.039683 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.041835 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.041847 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:37.044486 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:37.044501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.046542 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.046549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:37.046764 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.046797 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:56:37.047934 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0619 18:56:37.048127 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13435,"mutex_wait_us":1,"thread_start_us":11968,"threads_started":1}
I20260619 18:56:37.050236 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.037507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38637 (local address 127.0.0.1:56786)
0619 18:56:37.040432 (+ 2925us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:37.040457 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:37.040614 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:37.041467 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:37.041472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:37.041505 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:37.042036 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:37.042054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.044159 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.044166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:37.046427 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:37.046443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.049376 (+ 2933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.049383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:37.049583 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.049619 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:56:37.049758 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 18:56:37.049904 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2750,"thread_start_us":2566,"threads_started":1}
I20260619 18:56:37.050319 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.037548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.15.91.2:38637)
0619 18:56:37.039725 (+ 2177us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:37.039739 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:37.040760 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 18:56:37.040882 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:37.040892 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:37.041232 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:37.041463 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:37.042261 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.043968 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.046656 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.048859 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.048997 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.049200 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:37.049810 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 18:56:37.049986 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1935,"thread_start_us":1674,"threads_started":1}
I20260619 18:56:37.052812 30551 tablet_service.cc:1467] Processing CreateTablet for tablet 04c0eb258ba14d60b55dd7b3264aa21b (DEFAULT_TABLE table=test-table [id=2f667989a90548989025b4e4c5a76b2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:37.052814 30287 tablet_service.cc:1467] Processing CreateTablet for tablet 04c0eb258ba14d60b55dd7b3264aa21b (DEFAULT_TABLE table=test-table [id=2f667989a90548989025b4e4c5a76b2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:37.054071 30419 tablet_service.cc:1467] Processing CreateTablet for tablet 04c0eb258ba14d60b55dd7b3264aa21b (DEFAULT_TABLE table=test-table [id=2f667989a90548989025b4e4c5a76b2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:37.054548 30287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c0eb258ba14d60b55dd7b3264aa21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:37.054536 30551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c0eb258ba14d60b55dd7b3264aa21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:37.055413 30419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c0eb258ba14d60b55dd7b3264aa21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:37.070150 30633 tablet_bootstrap.cc:492] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: Bootstrap starting.
I20260619 18:56:37.072705 30634 tablet_bootstrap.cc:492] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: Bootstrap starting.
I20260619 18:56:37.074782 30633 tablet_bootstrap.cc:654] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:37.075192 30635 tablet_bootstrap.cc:492] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: Bootstrap starting.
I20260619 18:56:37.077266 30634 tablet_bootstrap.cc:654] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:37.078019 30633 log.cc:824] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:37.078789 30634 log.cc:824] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:37.079455 30635 tablet_bootstrap.cc:654] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:37.080884 30635 log.cc:824] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:37.081701 30633 tablet_bootstrap.cc:492] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: No bootstrap required, opened a new log
I20260619 18:56:37.082051 30633 ts_tablet_manager.cc:1397] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260619 18:56:37.082284 30634 tablet_bootstrap.cc:492] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: No bootstrap required, opened a new log
I20260619 18:56:37.082592 30634 ts_tablet_manager.cc:1397] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260619 18:56:37.084623 30635 tablet_bootstrap.cc:492] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: No bootstrap required, opened a new log
I20260619 18:56:37.084992 30635 ts_tablet_manager.cc:1397] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 18:56:37.090488 30633 raft_consensus.cc:348] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } } peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } }
I20260619 18:56:37.091071 30633 raft_consensus.cc:374] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:37.091212 30633 raft_consensus.cc:729] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d5e886e7a24122a99700e940a8f212, State: Initialized, Role: FOLLOWER
I20260619 18:56:37.092020 30633 consensus_queue.cc:260] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } } peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } }
I20260619 18:56:37.094252 30484 heartbeater.cc:503] Master 127.15.91.62:38525 was elected leader, sending a full tablet report...
I20260619 18:56:37.095127 30633 ts_tablet_manager.cc:1428] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260619 18:56:37.094880 30634 raft_consensus.cc:348] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } } peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } }
I20260619 18:56:37.095422 30634 raft_consensus.cc:374] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:37.095520 30634 raft_consensus.cc:729] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10401e7ce78c44e78745b8f5ae4d13e2, State: Initialized, Role: FOLLOWER
I20260619 18:56:37.096280 30634 consensus_queue.cc:260] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } } peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } }
I20260619 18:56:37.097383 30635 raft_consensus.cc:348] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } } peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } }
I20260619 18:56:37.098050 30635 raft_consensus.cc:374] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:37.098282 30635 raft_consensus.cc:729] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbb2c3c3cb24b169b98958d6a2daf8f, State: Initialized, Role: FOLLOWER
I20260619 18:56:37.099359 30634 ts_tablet_manager.cc:1428] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: Time spent starting tablet: real 0.017s user 0.017s sys 0.001s
I20260619 18:56:37.099130 30635 consensus_queue.cc:260] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } } peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } }
I20260619 18:56:37.100445 30616 heartbeater.cc:503] Master 127.15.91.62:38525 was elected leader, sending a full tablet report...
I20260619 18:56:37.102320 30635 ts_tablet_manager.cc:1428] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260619 18:56:37.126889 30353 tablet.cc:2369] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:56:37.165256 30485 tablet.cc:2369] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:56:37.198797 30617 tablet.cc:2369] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:56:37.390200 30639 raft_consensus.cc:482] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:56:37.390547 30639 raft_consensus.cc:504] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } } peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } }
I20260619 18:56:37.392632 30639 leader_election.cc:288] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dbb2c3c3cb24b169b98958d6a2daf8f (127.15.91.3:33483), 10401e7ce78c44e78745b8f5ae4d13e2 (127.15.91.1:42253)
I20260619 18:56:37.399798 30629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.392995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:53543 (local address 127.15.91.3:33483)
0619 18:56:37.393252 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:37.393260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:37.393879 (+ 619us) server_negotiation.cc:407] Connection header received
0619 18:56:37.394126 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:37.394133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:37.394207 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:37.394316 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:37.394964 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.395702 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.397839 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.398684 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.398909 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.399037 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:37.399370 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 18:56:37.399511 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:56:37.399963 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.392916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:33483 (local address 127.15.91.2:53543)
0619 18:56:37.393736 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:37.393756 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:37.393972 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:37.394334 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:37.394339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:37.394386 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:37.394792 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:37.394812 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.395862 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.395869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:37.397613 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:37.397631 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.398999 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.399006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:37.399179 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.399205 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:56:37.399322 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 18:56:37.399496 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":36,"thread_start_us":400,"threads_started":1}
I20260619 18:56:37.400175 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.393780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:60269 (local address 127.15.91.1:42253)
0619 18:56:37.393994 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:37.394002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:37.394075 (+ 73us) server_negotiation.cc:407] Connection header received
0619 18:56:37.394383 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:37.394389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:37.394463 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:37.394518 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:37.395299 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.396102 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.398262 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.399160 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.399324 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.399445 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:37.399772 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 18:56:37.399914 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 18:56:37.400297 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.393035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42253 (local address 127.15.91.2:60269)
0619 18:56:37.393967 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:37.393986 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:37.394198 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:37.394692 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:37.394698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:37.394781 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:37.395130 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:37.395139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.396263 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.396273 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:37.398077 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:37.398094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.399405 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.399411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:37.399599 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.399625 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:56:37.399742 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 18:56:37.399890 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":1018,"threads_started":1}
I20260619 18:56:37.401207 30571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c0eb258ba14d60b55dd7b3264aa21b" candidate_uuid: "13d5e886e7a24122a99700e940a8f212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" is_pre_election: true
I20260619 18:56:37.401292 30307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c0eb258ba14d60b55dd7b3264aa21b" candidate_uuid: "13d5e886e7a24122a99700e940a8f212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" is_pre_election: true
I20260619 18:56:37.401716 30571 raft_consensus.cc:2454] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d5e886e7a24122a99700e940a8f212 in term 0.
I20260619 18:56:37.401911 30307 raft_consensus.cc:2454] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d5e886e7a24122a99700e940a8f212 in term 0.
I20260619 18:56:37.402527 30373 leader_election.cc:302] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d5e886e7a24122a99700e940a8f212, 3dbb2c3c3cb24b169b98958d6a2daf8f; no voters:
I20260619 18:56:37.403091 30639 raft_consensus.cc:2781] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:56:37.403249 30639 raft_consensus.cc:482] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:56:37.403426 30639 raft_consensus.cc:3037] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:37.405838 30639 raft_consensus.cc:504] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } } peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } }
I20260619 18:56:37.406582 30639 leader_election.cc:288] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [CANDIDATE]: Term 1 election: Requested vote from peers 3dbb2c3c3cb24b169b98958d6a2daf8f (127.15.91.3:33483), 10401e7ce78c44e78745b8f5ae4d13e2 (127.15.91.1:42253)
I20260619 18:56:37.407269 30571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c0eb258ba14d60b55dd7b3264aa21b" candidate_uuid: "13d5e886e7a24122a99700e940a8f212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f"
I20260619 18:56:37.407298 30307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c0eb258ba14d60b55dd7b3264aa21b" candidate_uuid: "13d5e886e7a24122a99700e940a8f212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10401e7ce78c44e78745b8f5ae4d13e2"
I20260619 18:56:37.407537 30571 raft_consensus.cc:3037] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:37.407546 30307 raft_consensus.cc:3037] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:37.410075 30307 raft_consensus.cc:2454] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d5e886e7a24122a99700e940a8f212 in term 1.
I20260619 18:56:37.410233 30571 raft_consensus.cc:2454] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d5e886e7a24122a99700e940a8f212 in term 1.
I20260619 18:56:37.410866 30374 leader_election.cc:302] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10401e7ce78c44e78745b8f5ae4d13e2, 13d5e886e7a24122a99700e940a8f212, 3dbb2c3c3cb24b169b98958d6a2daf8f; no voters:
I20260619 18:56:37.411393 30639 raft_consensus.cc:2781] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:56:37.411979 30639 raft_consensus.cc:686] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 1 LEADER]: Becoming Leader. State: Replica: 13d5e886e7a24122a99700e940a8f212, State: Running, Role: LEADER
I20260619 18:56:37.412631 30639 consensus_queue.cc:237] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } } peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } }
I20260619 18:56:37.419025 30166 catalog_manager.cc:5495] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d5e886e7a24122a99700e940a8f212 (127.15.91.2). New cstate: current_term: 1 leader_uuid: "13d5e886e7a24122a99700e940a8f212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d5e886e7a24122a99700e940a8f212" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38637 } health_report { overall_health: HEALTHY } } }
I20260619 18:56:37.520684 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.514196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.15.91.2:38637)
0619 18:56:37.514508 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:37.514515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:37.514635 (+ 120us) server_negotiation.cc:407] Connection header received
0619 18:56:37.515060 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:37.515068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:37.515202 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:37.515367 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:56:37.515799 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.516567 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.518143 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.518595 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.518857 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.519461 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:56:37.520048 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:56:37.520183 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:37.520356 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 18:56:37.520443 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260619 18:56:37.530508 30307 raft_consensus.cc:1261] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2 [term 1 FOLLOWER]: Refusing update from remote peer 13d5e886e7a24122a99700e940a8f212: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:56:37.530501 30571 raft_consensus.cc:1261] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 1 FOLLOWER]: Refusing update from remote peer 13d5e886e7a24122a99700e940a8f212: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:56:37.531867 30644 consensus_queue.cc:1035] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:56:37.532364 30639 consensus_queue.cc:1035] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10401e7ce78c44e78745b8f5ae4d13e2" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:56:37.554497 30649 mvcc.cc:205] Tried to move back new op lower bound from 7298643548272177152 to 7298643547806806016. Current Snapshot: MvccSnapshot[applied={T|T < 7298643548272177152}]
I20260619 18:56:37.564079 30167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:56:37.564437 30167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:56:37.568872 30167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Sending DeleteTablet for 3 replicas of tablet 04c0eb258ba14d60b55dd7b3264aa21b
I20260619 18:56:37.570274 30287 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c0eb258ba14d60b55dd7b3264aa21b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:37 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:33224
I20260619 18:56:37.570295 30551 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c0eb258ba14d60b55dd7b3264aa21b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:37 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:51642
I20260619 18:56:37.571179 30419 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c0eb258ba14d60b55dd7b3264aa21b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:37 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:56786
I20260619 18:56:37.572326 30660 tablet_replica.cc:330] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: stopping tablet replica
I20260619 18:56:37.572669 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 30204
I20260619 18:56:37.572978 30660 raft_consensus.cc:2229] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:56:37.573798 30660 raft_consensus.cc:2258] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:37.576846 30660 ts_tablet_manager.cc:1905] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:37.585966 30660 ts_tablet_manager.cc:1918] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:56:37.586297 30658 tablet_replica.cc:330] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: stopping tablet replica
I20260619 18:56:37.586339 30660 log.cc:1196] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/04c0eb258ba14d60b55dd7b3264aa21b
I20260619 18:56:37.586927 30658 raft_consensus.cc:2229] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:56:37.587237 30660 ts_tablet_manager.cc:1939] T 04c0eb258ba14d60b55dd7b3264aa21b P 13d5e886e7a24122a99700e940a8f212: Deleting consensus metadata
I20260619 18:56:37.587575 30658 raft_consensus.cc:2258] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:37.589502 30152 catalog_manager.cc:4841] TS 13d5e886e7a24122a99700e940a8f212 (127.15.91.2:38637): tablet 04c0eb258ba14d60b55dd7b3264aa21b (table test-table [id=2f667989a90548989025b4e4c5a76b2c]) successfully deleted
I20260619 18:56:37.590277 30658 ts_tablet_manager.cc:1905] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:37.592120 30659 tablet_replica.cc:330] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: stopping tablet replica
I20260619 18:56:37.592657 30659 raft_consensus.cc:2229] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:56:37.593289 30659 raft_consensus.cc:2258] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:37.595742 30659 ts_tablet_manager.cc:1905] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:37.599365 30658 ts_tablet_manager.cc:1918] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:56:37.599615 30658 log.cc:1196] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/04c0eb258ba14d60b55dd7b3264aa21b
I20260619 18:56:37.600278 30658 ts_tablet_manager.cc:1939] T 04c0eb258ba14d60b55dd7b3264aa21b P 3dbb2c3c3cb24b169b98958d6a2daf8f: Deleting consensus metadata
I20260619 18:56:37.602545 30152 catalog_manager.cc:4841] TS 3dbb2c3c3cb24b169b98958d6a2daf8f (127.15.91.3:33483): tablet 04c0eb258ba14d60b55dd7b3264aa21b (table test-table [id=2f667989a90548989025b4e4c5a76b2c]) successfully deleted
I20260619 18:56:37.604279 30659 ts_tablet_manager.cc:1918] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:56:37.604513 30659 log.cc:1196] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/04c0eb258ba14d60b55dd7b3264aa21b
I20260619 18:56:37.605041 30659 ts_tablet_manager.cc:1939] T 04c0eb258ba14d60b55dd7b3264aa21b P 10401e7ce78c44e78745b8f5ae4d13e2: Deleting consensus metadata
I20260619 18:56:37.607077 30153 catalog_manager.cc:4841] TS 10401e7ce78c44e78745b8f5ae4d13e2 (127.15.91.1:42253): tablet 04c0eb258ba14d60b55dd7b3264aa21b (table test-table [id=2f667989a90548989025b4e4c5a76b2c]) successfully deleted
I20260619 18:56:37.612514 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.573817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.15.91.1:42253)
0619 18:56:37.574027 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:37.574035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:37.591458 (+ 17423us) server_negotiation.cc:407] Connection header received
0619 18:56:37.591702 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:37.591711 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:37.591820 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:37.591927 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:37.594839 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.596120 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.598893 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.599568 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.599675 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.604208 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:37.604240 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:37.604268 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:37.604332 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:37.608898 (+ 4566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:37.609833 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:37.609844 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:37.609848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:37.610043 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:37.610528 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:37.610535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:37.610538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:37.611121 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:37.611475 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:37.611833 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 18:56:37.612099 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 18:56:37.613446 30347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:37.707237 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 30204
I20260619 18:56:37.727263 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 30356
I20260619 18:56:37.743968 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.728280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.15.91.2:38637)
0619 18:56:37.728501 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:37.728507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:37.728530 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:56:37.728721 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:37.728727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:37.728815 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:37.728940 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:37.730513 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.731463 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.733014 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.733493 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.733654 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.736834 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:37.736853 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:37.736869 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:37.736916 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:37.740549 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:37.741545 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:37.741557 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:37.741560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:37.741720 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:37.742183 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:37.742191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:37.742199 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:37.742629 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:37.743008 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:37.743407 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 18:56:37.743610 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 18:56:37.744738 30479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:37.865545 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 30356
I20260619 18:56:37.886691 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 30488
I20260619 18:56:37.903054 30629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:37.887637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.15.91.3:33483)
0619 18:56:37.887835 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:37.887839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:37.887860 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:56:37.887983 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:37.887987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:37.888073 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:37.888168 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:37.889837 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.890638 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.892413 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:37.892848 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:37.893013 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:37.896393 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:37.896413 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:37.896431 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:37.896479 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:37.899675 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:37.900632 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:37.900641 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:37.900643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:37.900806 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:37.901241 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:37.901247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:37.901250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:37.901660 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:37.901985 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:37.902367 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 18:56:37.902620 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:56:37.903842 30611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:37.997143 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 30488
I20260619 18:56:38.017212 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 30134
I20260619 18:56:38.018672 30197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:38.139721 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 30134
I20260619 18:56:38.163600 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38525
--webserver_interface=127.15.91.62
--webserver_port=35695
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:38525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:38.535950 30666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:38.536303 30666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:38.536458 30666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:38.544741 30666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:56:38.544907 30666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:38.545019 30666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:38.545121 30666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:56:38.545221 30666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:56:38.557126 30666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:38525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=35695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:38.558777 30666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:38.560340 30666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:38.574837 30674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:38.574880 30671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:38.574855 30672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:38.576076 30666 server_base.cc:1034] running on GCE node
I20260619 18:56:38.577100 30666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:38.578956 30666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:38.580273 30666 hybrid_clock.cc:648] HybridClock initialized: now 1781895398580225 us; error 65 us; skew 500 ppm
I20260619 18:56:38.580832 30666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:38 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895398, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:38.585136 30666 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:56:38.587258 30666 webserver.cc:460] Webserver started at http://127.15.91.62:35695/ using document root <none> and password file <none>
I20260619 18:56:38.588138 30666 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:38.588330 30666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:38.595681 30666 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 18:56:38.599481 30681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:38.601194 30666 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260619 18:56:38.601460 30666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "8d1826f93b9c45d78e8da3ec2c9500e3"
format_stamp: "Formatted at 2026-06-19 18:56:31 on dist-test-slave-dm3f"
server_key: "94bb0c96327e1572bdeb0a4268f33582"
server_key_iv: "286f0b236c91fae149f2c1ce26f3b14a"
server_key_version: "encryptionkey@0"
I20260619 18:56:38.602620 30666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:38.617952 30666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:38.623118 30666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:38.623450 30666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:38.645460 30666 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:38525
I20260619 18:56:38.645466 30732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:38525 every 8 connection(s)
I20260619 18:56:38.647351 30666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:56:38.649986 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 30666
I20260619 18:56:38.651090 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:42253
--local_ip_for_outbound_sockets=127.15.91.1
--tserver_master_addrs=127.15.91.62:38525
--webserver_port=34387
--webserver_interface=127.15.91.1
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:56:38.658939 30733 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:56:38.663203 30733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Bootstrap starting.
I20260619 18:56:38.685823 30733 log.cc:824] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:38.705314 30733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:56:38.705996 30733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3: Bootstrap complete.
I20260619 18:56:38.716652 30733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:38.717700 30733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1826f93b9c45d78e8da3ec2c9500e3, State: Initialized, Role: FOLLOWER
I20260619 18:56:38.718532 30733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:38.718878 30733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:56:38.719027 30733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:56:38.719252 30733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:56:38.723920 30733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:38.724577 30733 leader_election.cc:302] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d1826f93b9c45d78e8da3ec2c9500e3; no voters:
I20260619 18:56:38.725385 30733 leader_election.cc:288] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 18:56:38.725538 30737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 18:56:38.726962 30737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [term 3 LEADER]: Becoming Leader. State: Replica: 8d1826f93b9c45d78e8da3ec2c9500e3, State: Running, Role: LEADER
I20260619 18:56:38.727686 30737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } }
I20260619 18:56:38.728129 30733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:56:38.731248 30738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } } }
I20260619 18:56:38.731240 30739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1826f93b9c45d78e8da3ec2c9500e3. Latest consensus state: current_term: 3 leader_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1826f93b9c45d78e8da3ec2c9500e3" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38525 } } }
I20260619 18:56:38.731913 30739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:38.731922 30738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1826f93b9c45d78e8da3ec2c9500e3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:38.745525 30744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:56:38.756623 30744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:56:38.759138 30744 catalog_manager.cc:1259] Loaded cluster ID: 1fd880b9a7874d11a97e805c4b0992e9
I20260619 18:56:38.759341 30744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 18:56:38.762751 30744 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f69ea4579ad google::LogMessage::Fail() at ??:0
@ 0x7f69ea45a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f69ea4574de google::LogMessage::Flush() at ??:0
@ 0x7f69ea458a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f69ffe8a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f69ffe87976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f69fff0f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f69fff0f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f69ed0a3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f69ed0a691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f69ed0a6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f69ed081eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f69f035b6db start_thread at ??:0
@ 0x7f69e92f161f clone at ??:0
W20260619 18:56:39.103899 30735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:39.104331 30735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:39.104532 30735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:39.113209 30735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:39.113410 30735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:39.113602 30735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:56:39.124655 30735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:42253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=34387
--tserver_master_addrs=127.15.91.62:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:39.126171 30735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:39.127789 30735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:39.144300 30775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:39.144271 30774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:39.145172 30777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:39.145464 30735 server_base.cc:1034] running on GCE node
I20260619 18:56:39.146214 30735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:39.147498 30735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:39.148813 30735 hybrid_clock.cc:648] HybridClock initialized: now 1781895399148742 us; error 64 us; skew 500 ppm
I20260619 18:56:39.149281 30735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:39 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895399, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:39.153136 30735 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:56:39.155124 30735 webserver.cc:460] Webserver started at http://127.15.91.1:34387/ using document root <none> and password file <none>
I20260619 18:56:39.155927 30735 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:39.156060 30735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:39.162864 30735 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 18:56:39.166428 30784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:39.168761 30735 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:56:39.169057 30735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "10401e7ce78c44e78745b8f5ae4d13e2"
format_stamp: "Formatted at 2026-06-19 18:56:32 on dist-test-slave-dm3f"
server_key: "32a944f9ef6311d620f1f3c3a402a120"
server_key_iv: "a47c58e081d962aa746cee2f25732124"
server_key_version: "encryptionkey@0"
I20260619 18:56:39.170303 30735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:39.194384 30735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:39.200140 30735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:39.200520 30735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:39.201820 30735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:39.203642 30735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:39.203778 30735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:39.203979 30735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:39.204087 30735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:39.244832 30735 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:42253
I20260619 18:56:39.244900 30896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:42253 every 8 connection(s)
I20260619 18:56:39.246577 30735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:56:39.252614 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:39.250255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:39173)
0619 18:56:39.251275 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:39.251527 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":475,"threads_started":1}
W20260619 18:56:39.253532 30897 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:38525 (0 consecutive failures): Network error: Failed to ping master at 127.15.91.62:38525: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
I20260619 18:56:39.254297 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 30735
I20260619 18:56:39.254809 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:39.254288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:48321)
0619 18:56:39.254484 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:39.254595 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 18:56:39.255187 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:38637
--local_ip_for_outbound_sockets=127.15.91.2
--tserver_master_addrs=127.15.91.62:38525
--webserver_port=44929
--webserver_interface=127.15.91.2
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:56:39.256012 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:39.255591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:54769)
0619 18:56:39.255785 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:39.255872 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 18:56:39.256331 30897 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:56:39.627408 30900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:39.627735 30900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:39.627923 30900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:39.635995 30900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:39.636150 30900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:39.636353 30900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:56:39.647087 30900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:38637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=44929
--tserver_master_addrs=127.15.91.62:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:39.648630 30900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:39.650152 30900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:39.665380 30905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:39.665328 30906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:39.665642 30908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:39.666383 30900 server_base.cc:1034] running on GCE node
I20260619 18:56:39.667279 30900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:39.668412 30900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:39.669665 30900 hybrid_clock.cc:648] HybridClock initialized: now 1781895399669579 us; error 78 us; skew 500 ppm
I20260619 18:56:39.670063 30900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:39 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895399, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:39.673794 30900 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:56:39.676102 30900 webserver.cc:460] Webserver started at http://127.15.91.2:44929/ using document root <none> and password file <none>
I20260619 18:56:39.676923 30900 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:39.677055 30900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:39.684056 30900 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:56:39.687529 30915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:39.689378 30900 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:56:39.689657 30900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "13d5e886e7a24122a99700e940a8f212"
format_stamp: "Formatted at 2026-06-19 18:56:33 on dist-test-slave-dm3f"
server_key: "aa88148a984ec9042a36ac87b6cf945d"
server_key_iv: "8cc7401106df85cb083c5136e60b8d61"
server_key_version: "encryptionkey@0"
I20260619 18:56:39.690888 30900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:39.715199 30900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:39.720508 30900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:39.720878 30900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:39.722108 30900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:39.723815 30900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:39.723986 30900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:39.724161 30900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:39.724257 30900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:39.765038 30900 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:38637
I20260619 18:56:39.765103 31027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:38637 every 8 connection(s)
I20260619 18:56:39.766896 30900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:56:39.772791 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:39.770448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:38103)
0619 18:56:39.771448 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:39.771704 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":457,"threads_started":1}
W20260619 18:56:39.773766 31028 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:38525 (0 consecutive failures): Network error: Failed to ping master at 127.15.91.62:38525: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
I20260619 18:56:39.774237 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 30900
I20260619 18:56:39.775039 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:33483
--local_ip_for_outbound_sockets=127.15.91.3
--tserver_master_addrs=127.15.91.62:38525
--webserver_port=35113
--webserver_interface=127.15.91.3
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:56:39.775177 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:39.774607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:40371)
0619 18:56:39.774845 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:39.774976 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 18:56:39.776973 31031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:39.775882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:45449)
0619 18:56:39.776594 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:39.776732 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":357,"threads_started":1}
W20260619 18:56:39.777413 31028 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:56:40.139878 31032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:40.140218 31032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:40.140401 31032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:40.148638 31032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:40.148803 31032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:40.149036 31032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:56:40.159696 31032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:33483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=35113
--tserver_master_addrs=127.15.91.62:38525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:40.161211 31032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:40.162801 31032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:40.178871 31038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:40.179103 31040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:40.179371 31037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:40.180359 31032 server_base.cc:1034] running on GCE node
I20260619 18:56:40.181118 31032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:40.182262 31032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:40.183547 31032 hybrid_clock.cc:648] HybridClock initialized: now 1781895400183500 us; error 67 us; skew 500 ppm
I20260619 18:56:40.183966 31032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:40 dist-test-slave-dm3f krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895400, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:40.187695 31032 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:56:40.189695 31032 webserver.cc:460] Webserver started at http://127.15.91.3:35113/ using document root <none> and password file <none>
I20260619 18:56:40.190521 31032 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:40.190716 31032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:40.198266 31032 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260619 18:56:40.202409 31047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:40.204391 31032 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:56:40.204641 31032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "3dbb2c3c3cb24b169b98958d6a2daf8f"
format_stamp: "Formatted at 2026-06-19 18:56:33 on dist-test-slave-dm3f"
server_key: "72eaa927a53db975a54a9168576c072a"
server_key_iv: "6a11157529a7112d86074083fc2cc280"
server_key_version: "encryptionkey@0"
I20260619 18:56:40.205816 31032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:40.231082 31032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:40.236500 31032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:40.236915 31032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:40.238304 31032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:40.240123 31032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:40.240248 31032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:40.240397 31032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:40.240473 31032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:40.258080 31095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:40.257104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:43745)
0619 18:56:40.257704 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:40.257822 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":302,"threads_started":1}
I20260619 18:56:40.281112 31032 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:33483
I20260619 18:56:40.281231 31160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:33483 every 8 connection(s)
I20260619 18:56:40.282886 31032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:56:40.283241 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 31032
I20260619 18:56:40.289135 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:40.286662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:58933)
0619 18:56:40.287788 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:40.288042 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":365,"threads_started":1}
W20260619 18:56:40.290100 31161 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:38525 (0 consecutive failures): Network error: Failed to ping master at 127.15.91.62:38525: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
I20260619 18:56:40.291333 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:40.290911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:54929)
0619 18:56:40.291105 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:40.291191 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 18:56:40.292477 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:40.292009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:56757)
0619 18:56:40.292236 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:40.292331 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260619 18:56:40.292801 31161 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:56:40.779163 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:40.778182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:57039)
0619 18:56:40.778798 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:40.778912 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":263,"threads_started":1}
I20260619 18:56:41.260141 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:41.259262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:55627)
0619 18:56:41.259837 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:41.259938 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":274,"threads_started":1}
I20260619 18:56:41.295028 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:41.293734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:46803)
0619 18:56:41.294458 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:41.294680 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":356,"threads_started":1}
I20260619 18:56:41.781177 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:41.780399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:53403)
0619 18:56:41.780897 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:41.781000 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":250,"threads_started":1}
I20260619 18:56:42.262377 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:42.261422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:39319)
0619 18:56:42.262012 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:42.262113 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":357,"threads_started":1}
I20260619 18:56:42.297425 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:42.296487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:40423)
0619 18:56:42.297152 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:42.297243 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":391,"threads_started":1}
I20260619 18:56:42.783151 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:42.782266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:49483)
0619 18:56:42.782842 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:42.782940 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":253,"threads_started":1}
I20260619 18:56:43.264417 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:43.263569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:34359)
0619 18:56:43.264109 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:43.264213 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":285,"threads_started":1}
I20260619 18:56:43.299467 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:43.298544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:57725)
0619 18:56:43.299144 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:43.299260 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":372,"threads_started":1}
I20260619 18:56:43.785441 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:43.784544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:38635)
0619 18:56:43.785129 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:43.785263 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":293,"threads_started":1}
I20260619 18:56:44.266434 31175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:44.265590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:56627)
0619 18:56:44.266135 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:44.266241 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":291,"threads_started":1}
I20260619 18:56:44.301635 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:44.300814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:40235)
0619 18:56:44.301332 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:44.301429 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":257,"threads_started":1}
I20260619 18:56:44.787786 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:44.786853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:42901)
0619 18:56:44.787480 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:44.787578 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":344,"threads_started":1}
I20260619 18:56:45.268510 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:45.267625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:36985)
0619 18:56:45.268210 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:45.268325 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":326,"threads_started":1}
I20260619 18:56:45.303815 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:45.302913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:54493)
0619 18:56:45.303524 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:45.303618 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":334,"threads_started":1}
I20260619 18:56:45.790053 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:45.789124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:57085)
0619 18:56:45.789753 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:45.789876 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":277,"threads_started":1}
I20260619 18:56:46.270478 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:46.269561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:56135)
0619 18:56:46.270142 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:46.270239 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":269,"threads_started":1}
I20260619 18:56:46.305763 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:46.304841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:53663)
0619 18:56:46.305471 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:46.305574 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":367,"threads_started":1}
I20260619 18:56:46.792383 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:46.791323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:47465)
0619 18:56:46.791960 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:46.792107 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":287,"threads_started":1}
I20260619 18:56:47.272441 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:47.271659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:56535)
0619 18:56:47.272156 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:47.272258 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":257,"threads_started":1}
I20260619 18:56:47.308095 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:47.307087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:46631)
0619 18:56:47.307751 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:47.307852 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":321,"threads_started":1}
I20260619 18:56:47.794204 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:47.793442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:51005)
0619 18:56:47.793922 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:47.794016 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":242,"threads_started":1}
I20260619 18:56:48.274878 31187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:48.273773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:44185)
0619 18:56:48.274402 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:48.274500 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":383,"threads_started":1}
I20260619 18:56:48.310600 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:48.309585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:58391)
0619 18:56:48.310201 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:48.310420 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":308,"threads_started":1}
I20260619 18:56:48.796309 31189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:48.795427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:56113)
0619 18:56:48.795977 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:48.796087 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":277,"threads_started":1}
I20260619 18:56:49.277365 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:49.276280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:45035)
0619 18:56:49.277005 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:49.277143 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":420,"threads_started":1}
I20260619 18:56:49.312481 31191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:49.311602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:49411)
0619 18:56:49.312172 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:49.312281 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":258,"threads_started":1}
I20260619 18:56:49.798291 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:49.797440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:41301)
0619 18:56:49.797989 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:49.798094 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":282,"threads_started":1}
I20260619 18:56:50.279826 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:50.278903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:52385)
0619 18:56:50.279437 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:50.279596 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":254,"threads_started":1}
I20260619 18:56:50.314708 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:50.313722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:60757)
0619 18:56:50.314334 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:50.314471 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":323,"threads_started":1}
I20260619 18:56:50.800129 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:50.799360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:34725)
0619 18:56:50.799836 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:50.799937 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":232,"threads_started":1}
I20260619 18:56:51.282373 31196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:51.281073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:40341)
0619 18:56:51.281847 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:51.282135 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":258,"threads_started":1}
I20260619 18:56:51.316840 31197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:51.315989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:49257)
0619 18:56:51.316494 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:51.316610 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":254,"threads_started":1}
I20260619 18:56:51.802259 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:51.801290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:58553)
0619 18:56:51.801900 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:51.801998 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":325,"threads_started":1}
I20260619 18:56:52.284556 31199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:52.283577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:56855)
0619 18:56:52.284219 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:52.284358 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":353,"threads_started":1}
I20260619 18:56:52.318874 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:52.317974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:54443)
0619 18:56:52.318514 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:52.318611 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
I20260619 18:56:52.804322 31201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:52.803430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:43317)
0619 18:56:52.804011 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:52.804110 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":294,"threads_started":1}
I20260619 18:56:53.286820 31202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:53.285632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:54205)
0619 18:56:53.286432 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:53.286531 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":583,"threads_started":1}
I20260619 18:56:53.321429 31203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:53.320101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:58277)
0619 18:56:53.320837 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:53.321174 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":346,"threads_started":1}
I20260619 18:56:53.806380 31204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:53.805427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:36529)
0619 18:56:53.806035 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:53.806143 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":390,"threads_started":1}
I20260619 18:56:54.289301 31205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:54.288314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:42159)
0619 18:56:54.288924 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:54.289050 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":327,"threads_started":1}
I20260619 18:56:54.323619 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:54.322615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:54087)
0619 18:56:54.323266 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:54.323415 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":390,"threads_started":1}
I20260619 18:56:54.808689 31207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:54.807829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.2:59197)
0619 18:56:54.808349 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:54.808485 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":271,"threads_started":1}
I20260619 18:56:55.286784 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 30735
I20260619 18:56:55.291718 31209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:55.290332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.1:49729)
0619 18:56:55.291084 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:55.291394 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":350,"threads_started":1}
I20260619 18:56:55.315059 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:55.288186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.15.91.1:42253)
0619 18:56:55.289002 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:55.289024 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:55.289078 (+ 54us) server_negotiation.cc:407] Connection header received
0619 18:56:55.291867 (+ 2789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:55.291920 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:55.292518 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:55.292895 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:55.294860 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:55.296763 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:55.298513 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:55.299926 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:55.300158 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:55.304253 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:55.304265 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:55.304286 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:55.304340 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:55.308504 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:55.309674 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:55.309685 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:55.309697 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:55.310104 (+ 407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:55.310708 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:55.310741 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:55.310744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:55.311739 (+ 995us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:55.312092 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:55.313706 (+ 1614us) server_negotiation.cc:299] Negotiation successful
0619 18:56:55.314324 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":360,"threads_started":1}
I20260619 18:56:55.317416 30892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:55.326512 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:55.325160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:38525 (local address 127.15.91.3:54149)
0619 18:56:55.325905 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:55.326139 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":322,"threads_started":1}
I20260619 18:56:55.416571 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 30735
I20260619 18:56:55.436230 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 30900
I20260619 18:56:55.459506 31212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:55.437814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.15.91.2:38637)
0619 18:56:55.438529 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:55.438546 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:55.438590 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:56:55.440262 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:55.440274 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:55.440780 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:55.441129 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:55.443255 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:55.444677 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:55.446481 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:55.447426 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:55.447620 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:55.451082 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:55.451092 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:55.451113 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:55.451174 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:55.455192 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:55.456236 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:55.456246 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:55.456254 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:55.456484 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:55.457050 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:55.457054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:55.457057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:55.457777 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:55.458084 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:55.458559 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 18:56:55.459158 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":320,"threads_started":1}
I20260619 18:56:55.461225 31023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:55.551779 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 30900
I20260619 18:56:55.572038 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 31032
I20260619 18:56:55.594576 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:55.573268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.15.91.3:33483)
0619 18:56:55.574238 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:55.574255 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:55.574307 (+ 52us) server_negotiation.cc:407] Connection header received
0619 18:56:55.576372 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:55.576386 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:55.576961 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:55.577414 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:55.579303 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:55.580655 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:55.582163 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:55.583058 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:55.583304 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:55.586364 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:55.586374 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:55.586393 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:55.586447 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:55.590299 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:55.591445 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:55.591454 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:55.591464 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:55.591700 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:55.592166 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:55.592172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:55.592177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:55.592830 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:55.593200 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:55.593707 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 18:56:55.594240 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":426,"threads_started":1}
I20260619 18:56:55.597075 31156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:56:55.687397 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 31032
2026-06-19T18:56:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25637 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:56:55 dist-test-slave-dm3f krb5kdc[31218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:56:55 dist-test-slave-dm3f krb5kdc[31218](info): set up 2 sockets
Jun 19 18:56:55 dist-test-slave-dm3f krb5kdc[31218](info): commencing operation
krb5kdc: starting...
W20260619 18:56:56.803658 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:56:56 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:56:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:56:57.067910 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46783
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:57.439116 31231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:57.439458 31231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:57.439651 31231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:57.447729 31231 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:56:57.447867 31231 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:56:57.447907 31231 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:56:57.447984 31231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:56:57.448071 31231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:57.448194 31231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:57.448282 31231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:56:57.448374 31231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:56:57.459554 31231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:57.461012 31231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:57.462597 31231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:57.477083 31239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:57.477083 31237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:57.477348 31231 server_base.cc:1034] running on GCE node
W20260619 18:56:57.477448 31236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:57.478575 31231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:57.480360 31231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:57.481690 31231 hybrid_clock.cc:648] HybridClock initialized: now 1781895417481621 us; error 52 us; skew 500 ppm
I20260619 18:56:57.482138 31231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:57 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895417, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:57.489661 31231 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:56:57.491725 31231 webserver.cc:460] Webserver started at http://127.15.91.62:41145/ using document root <none> and password file <none>
I20260619 18:56:57.492643 31231 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:57.492836 31231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:57.493211 31231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:57.496840 31231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "6e0bb6a0349e44c79a22775ced98821e"
format_stamp: "Formatted at 2026-06-19 18:56:57 on dist-test-slave-dm3f"
server_key: "f8fbb66561950b6af3c4d38862092bd8"
server_key_iv: "fee797a3811fc4ce654d15563f6ad48b"
server_key_version: "encryptionkey@0"
I20260619 18:56:57.497889 31231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "6e0bb6a0349e44c79a22775ced98821e"
format_stamp: "Formatted at 2026-06-19 18:56:57 on dist-test-slave-dm3f"
server_key: "f8fbb66561950b6af3c4d38862092bd8"
server_key_iv: "fee797a3811fc4ce654d15563f6ad48b"
server_key_version: "encryptionkey@0"
I20260619 18:56:57.504559 31231 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260619 18:56:57.508917 31246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:57.510550 31231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:56:57.510833 31231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "6e0bb6a0349e44c79a22775ced98821e"
format_stamp: "Formatted at 2026-06-19 18:56:57 on dist-test-slave-dm3f"
server_key: "f8fbb66561950b6af3c4d38862092bd8"
server_key_iv: "fee797a3811fc4ce654d15563f6ad48b"
server_key_version: "encryptionkey@0"
I20260619 18:56:57.511127 31231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:57.538175 31231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:57.543541 31231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:57.543895 31231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:57.568123 31231 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:46783
I20260619 18:56:57.568142 31297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:46783 every 8 connection(s)
I20260619 18:56:57.569768 31231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:56:57.574380 31298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:57.576412 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 31231
I20260619 18:56:57.576710 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:56:57.577548 15724 external_mini_cluster.cc:1428] Setting key d2d19c4f4bbf2140d9eef9a2482301f2
I20260619 18:56:57.589733 31298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Bootstrap starting.
I20260619 18:56:57.593675 31298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:57.595453 31298 log.cc:824] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:56:57 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:57.599684 31298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: No bootstrap required, opened a new log
I20260619 18:56:57.610600 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:57.580286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.15.91.62:46783)
0619 18:56:57.581584 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:57.581609 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:57.581695 (+ 86us) server_negotiation.cc:407] Connection header received
0619 18:56:57.584202 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:57.584245 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:57.584913 (+ 668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:57.585682 (+ 769us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:57.587609 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:57.589485 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:57.591710 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:57.592639 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:57.592946 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:57.598918 (+ 5972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:57.598928 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:57.598947 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:57.598995 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:57.603784 (+ 4789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:57.604966 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:57.604977 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:57.604987 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:57.605260 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:57.606224 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:57.606230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:57.606234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:57.607250 (+ 1016us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:57.607652 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:57.608320 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 18:56:57.608947 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":533,"threads_started":1}
I20260619 18:56:57.611478 31298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:56:57.612010 31298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:57.612166 31298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0bb6a0349e44c79a22775ced98821e, State: Initialized, Role: FOLLOWER
I20260619 18:56:57.612952 31298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:56:57.613258 31298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:56:57.613418 31298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:56:57.613641 31298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:57.616636 31298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:56:57.617349 31298 leader_election.cc:302] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e0bb6a0349e44c79a22775ced98821e; no voters:
I20260619 18:56:57.618073 31298 leader_election.cc:288] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:56:57.618260 31303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:56:57.619253 31303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 1 LEADER]: Becoming Leader. State: Replica: 6e0bb6a0349e44c79a22775ced98821e, State: Running, Role: LEADER
I20260619 18:56:57.619968 31303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:56:57.620775 31298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:56:57.623500 31304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e0bb6a0349e44c79a22775ced98821e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } } }
I20260619 18:56:57.623605 31305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e0bb6a0349e44c79a22775ced98821e. Latest consensus state: current_term: 1 leader_uuid: "6e0bb6a0349e44c79a22775ced98821e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } } }
I20260619 18:56:57.623984 31304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:57.624068 31305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: This master's current role is: LEADER
I20260619 18:56:57.626869 31309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:56:57.633284 31309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:56:57.645108 31309 catalog_manager.cc:1347] Generated new cluster ID: dc3411a815d14756b38280012ecaafb0
I20260619 18:56:57.645251 31309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:56:57.661984 31309 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:56:57.663616 31309 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:56:57.672209 31309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Generated new TSK 0
I20260619 18:56:57.673225 31309 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:56:57.742596 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46783
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:58.114168 31324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:58.114496 31324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:58.114599 31324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:58.123144 31324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:58.123287 31324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:58.123412 31324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:56:58.134835 31324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:58.136475 31324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:58.138015 31324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:58.153383 31330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:58.153407 31332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:58.153371 31329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:58.154376 31324 server_base.cc:1034] running on GCE node
I20260619 18:56:58.155180 31324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:58.156360 31324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:58.157601 31324 hybrid_clock.cc:648] HybridClock initialized: now 1781895418157527 us; error 66 us; skew 500 ppm
I20260619 18:56:58.158020 31324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:58 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895418, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:58.161744 31324 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:56:58.163565 31324 webserver.cc:460] Webserver started at http://127.15.91.1:39981/ using document root <none> and password file <none>
I20260619 18:56:58.164383 31324 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:58.164515 31324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:58.164803 31324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:58.168192 31324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "8e68a91804f3473aa67cfacd693e4c58"
format_stamp: "Formatted at 2026-06-19 18:56:58 on dist-test-slave-dm3f"
server_key: "d640d9397d4c3bc60c5e724a6481a7d9"
server_key_iv: "cd9144fa33a77d3b27bbce384329f986"
server_key_version: "encryptionkey@0"
I20260619 18:56:58.169090 31324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "8e68a91804f3473aa67cfacd693e4c58"
format_stamp: "Formatted at 2026-06-19 18:56:58 on dist-test-slave-dm3f"
server_key: "d640d9397d4c3bc60c5e724a6481a7d9"
server_key_iv: "cd9144fa33a77d3b27bbce384329f986"
server_key_version: "encryptionkey@0"
I20260619 18:56:58.175522 31324 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 18:56:58.179719 31339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:58.181416 31324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:56:58.181608 31324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "8e68a91804f3473aa67cfacd693e4c58"
format_stamp: "Formatted at 2026-06-19 18:56:58 on dist-test-slave-dm3f"
server_key: "d640d9397d4c3bc60c5e724a6481a7d9"
server_key_iv: "cd9144fa33a77d3b27bbce384329f986"
server_key_version: "encryptionkey@0"
I20260619 18:56:58.181823 31324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:58.197746 31324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:58.202997 31324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:58.203297 31324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:58.204609 31324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:58.206308 31324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:58.206408 31324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:58.206512 31324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:58.206562 31324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:58.253182 31324 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:42901
I20260619 18:56:58.253374 31451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:42901 every 8 connection(s)
I20260619 18:56:58.254769 31324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:56:58.260067 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 31324
I20260619 18:56:58.260336 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:56:58.261137 15724 external_mini_cluster.cc:1428] Setting key fc6af313576611ec267458604eab8df3
Jun 19 18:56:58 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895418, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:58.280298 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:58.258318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:49795 (local address 127.15.91.62:46783)
0619 18:56:58.258905 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:58.258910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:58.260812 (+ 1902us) server_negotiation.cc:407] Connection header received
0619 18:56:58.263054 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:58.263060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:58.263172 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:58.263310 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:58.265252 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:58.266003 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:58.268069 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:58.268520 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:58.268694 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:58.273384 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:58.273390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:58.273393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:58.273434 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:58.276468 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:58.277529 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:58.277534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:58.277537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:58.277685 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:58.278233 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:58.278238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:58.278240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:58.278549 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:58.278731 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:58.279607 (+ 876us) server_negotiation.cc:299] Negotiation successful
0619 18:56:58.279893 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":316,"threads_started":1}
I20260619 18:56:58.281541 31455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:58.258919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:49795)
0619 18:56:58.260566 (+ 1647us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:58.260628 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:58.262619 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:58.263617 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:58.263627 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:58.264287 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:58.264973 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:58.264995 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:58.266246 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:58.266251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:58.267808 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:58.267833 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:58.268837 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:58.268846 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:58.269128 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:58.269995 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:58.270022 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:58.273034 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:58.276737 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:58.276749 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:58.276778 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:58.277336 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:58.277821 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:58.277834 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:58.277838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:58.278085 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:58.278703 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:58.278735 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:58.279167 (+ 432us) client_negotiation.cc:769] Sending connection context
0619 18:56:58.279474 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 18:56:58.280115 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":589,"threads_started":1}
I20260619 18:56:58.283667 31452 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46783
I20260619 18:56:58.284117 31452 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:58.285264 31452 heartbeater.cc:511] Master 127.15.91.62:46783 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:56:58.289059 31263 ts_manager.cc:194] Registered new tserver with Master: 8e68a91804f3473aa67cfacd693e4c58 (127.15.91.1:42901)
I20260619 18:56:58.290805 31263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:49795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:56:58.318202 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46783
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:58.687894 31458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:58.688220 31458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:58.688388 31458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:58.696491 31458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:58.696641 31458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:58.696862 31458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:56:58.707763 31458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:58.709227 31458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:58.710785 31458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:56:58.726456 31466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:58.726392 31464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:58.726500 31463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:58.727524 31458 server_base.cc:1034] running on GCE node
I20260619 18:56:58.728344 31458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:58.729579 31458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:58.730851 31458 hybrid_clock.cc:648] HybridClock initialized: now 1781895418730784 us; error 61 us; skew 500 ppm
I20260619 18:56:58.731249 31458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:58 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895418, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:58.735190 31458 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:56:58.736985 31458 webserver.cc:460] Webserver started at http://127.15.91.2:43769/ using document root <none> and password file <none>
I20260619 18:56:58.737797 31458 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:58.737928 31458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:58.738220 31458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:58.741676 31458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "c47795cd4d3441a0b2f064543b30c4e1"
format_stamp: "Formatted at 2026-06-19 18:56:58 on dist-test-slave-dm3f"
server_key: "741b30708213ce6a2cfe55fce83bfb09"
server_key_iv: "15034a67d35a1b549fb72c593489c025"
server_key_version: "encryptionkey@0"
I20260619 18:56:58.742580 31458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "c47795cd4d3441a0b2f064543b30c4e1"
format_stamp: "Formatted at 2026-06-19 18:56:58 on dist-test-slave-dm3f"
server_key: "741b30708213ce6a2cfe55fce83bfb09"
server_key_iv: "15034a67d35a1b549fb72c593489c025"
server_key_version: "encryptionkey@0"
I20260619 18:56:58.749063 31458 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:56:58.753420 31473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:58.755577 31458 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:56:58.755836 31458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "c47795cd4d3441a0b2f064543b30c4e1"
format_stamp: "Formatted at 2026-06-19 18:56:58 on dist-test-slave-dm3f"
server_key: "741b30708213ce6a2cfe55fce83bfb09"
server_key_iv: "15034a67d35a1b549fb72c593489c025"
server_key_version: "encryptionkey@0"
I20260619 18:56:58.756158 31458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:58.785501 31458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:58.790994 31458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:58.791354 31458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:58.792532 31458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:58.794245 31458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:58.794396 31458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:58.794571 31458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:58.794653 31458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:58.834134 31458 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:38467
I20260619 18:56:58.834182 31585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:38467 every 8 connection(s)
I20260619 18:56:58.835719 31458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:56:58.836187 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 31458
I20260619 18:56:58.836410 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:56:58.837193 15724 external_mini_cluster.cc:1428] Setting key 5e311a5aa839e44006d47fd6c211d123
Jun 19 18:56:58 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895418, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:58.860085 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:58.839159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:37467 (local address 127.15.91.62:46783)
0619 18:56:58.839682 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:58.839688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:58.841055 (+ 1367us) server_negotiation.cc:407] Connection header received
0619 18:56:58.843194 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:58.843201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:58.843323 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:58.843453 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:58.845288 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:58.846070 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:58.848167 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:58.848674 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:58.848846 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:58.853381 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:58.853387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:58.853390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:58.853432 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:58.856355 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:58.857336 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:58.857342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:58.857344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:58.857502 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:58.858011 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:58.858023 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:58.858026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:58.858323 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:58.858493 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:58.859367 (+ 874us) server_negotiation.cc:299] Negotiation successful
0619 18:56:58.859655 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":294,"threads_started":1}
I20260619 18:56:58.861491 31589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:58.839924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:37467)
0619 18:56:58.840855 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:58.840907 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:58.842830 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:58.843751 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:58.843761 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:58.844395 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:58.845046 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:58.845066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:58.846273 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:58.846278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:58.847946 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:58.847970 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:58.848976 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:58.848984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:58.849270 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:58.850088 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:58.850113 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:58.853034 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:58.856574 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:58.856586 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:58.856614 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:58.857157 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:58.857635 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:58.857641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:58.857645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:58.857872 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:58.858482 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:58.858491 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:58.858959 (+ 468us) client_negotiation.cc:769] Sending connection context
0619 18:56:58.859265 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 18:56:58.859887 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":396,"threads_started":1}
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
I20260619 18:56:58.863817 31586 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46783
I20260619 18:56:58.864399 31586 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
I20260619 18:56:58.865419 31586 heartbeater.cc:511] Master 127.15.91.62:46783 requested a full tablet report, sending...
I20260619 18:56:58.867470 31263 ts_manager.cc:194] Registered new tserver with Master: c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
I20260619 18:56:58.868867 31263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:37467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:56:58.894665 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46783
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:56:59.262564 31592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:56:59.262976 31592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:56:59.263087 31592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:56:59.271167 31592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:56:59.271312 31592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:56:59.271435 31592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:56:59.282228 31592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:56:59.283771 31592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:56:59.285180 31592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:56:59.294682 31452 heartbeater.cc:503] Master 127.15.91.62:46783 was elected leader, sending a full tablet report...
W20260619 18:56:59.302493 31598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:59.302439 31600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:56:59.302532 31597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:56:59.304142 31592 server_base.cc:1034] running on GCE node
I20260619 18:56:59.304972 31592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:56:59.306294 31592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:56:59.307751 31592 hybrid_clock.cc:648] HybridClock initialized: now 1781895419307626 us; error 139 us; skew 500 ppm
I20260619 18:56:59.308223 31592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:56:59 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895419, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:56:59.312233 31592 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:56:59.314209 31592 webserver.cc:460] Webserver started at http://127.15.91.3:43849/ using document root <none> and password file <none>
I20260619 18:56:59.315259 31592 fs_manager.cc:359] Metadata directory not provided
I20260619 18:56:59.315398 31592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:56:59.315758 31592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:56:59.319353 31592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "3868e0574b874913bb7d7a0a0953fb39"
format_stamp: "Formatted at 2026-06-19 18:56:59 on dist-test-slave-dm3f"
server_key: "cd60e9c3febb477df37c7aff3417c503"
server_key_iv: "6548cc574ecc14e3dce9b5b0640b3926"
server_key_version: "encryptionkey@0"
I20260619 18:56:59.320387 31592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "3868e0574b874913bb7d7a0a0953fb39"
format_stamp: "Formatted at 2026-06-19 18:56:59 on dist-test-slave-dm3f"
server_key: "cd60e9c3febb477df37c7aff3417c503"
server_key_iv: "6548cc574ecc14e3dce9b5b0640b3926"
server_key_version: "encryptionkey@0"
I20260619 18:56:59.327273 31592 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260619 18:56:59.331990 31607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:59.333787 31592 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:56:59.334057 31592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "3868e0574b874913bb7d7a0a0953fb39"
format_stamp: "Formatted at 2026-06-19 18:56:59 on dist-test-slave-dm3f"
server_key: "cd60e9c3febb477df37c7aff3417c503"
server_key_iv: "6548cc574ecc14e3dce9b5b0640b3926"
server_key_version: "encryptionkey@0"
I20260619 18:56:59.334412 31592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:56:59.347720 31592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:56:59.353240 31592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:56:59.353622 31592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:56:59.354929 31592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:56:59.356796 31592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:56:59.356935 31592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:59.357100 31592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:56:59.357177 31592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:56:59.400789 31592 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:39201
I20260619 18:56:59.400837 31719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:39201 every 8 connection(s)
I20260619 18:56:59.402390 31592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:56:59.412175 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 31592
I20260619 18:56:59.412497 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:56:59.413053 15724 external_mini_cluster.cc:1428] Setting key e74ac3e9d4916d57d95650d51e3def29
Jun 19 18:56:59 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895419, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:56:59.428215 31722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.405683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:50867 (local address 127.15.91.62:46783)
0619 18:56:59.406562 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:59.406572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:59.407805 (+ 1233us) server_negotiation.cc:407] Connection header received
0619 18:56:59.409980 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:59.409987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:59.410112 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:59.410255 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:59.412352 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.413372 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.415927 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.416405 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.416572 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.421033 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:59.421040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:59.421043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:59.421087 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:59.424292 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:59.425336 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:59.425342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:59.425344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:59.425493 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:59.426018 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:59.426022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:59.426024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:59.426330 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:59.426516 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:59.427489 (+ 973us) server_negotiation.cc:299] Negotiation successful
0619 18:56:59.427786 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":590,"threads_started":1}
I20260619 18:56:59.429363 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.406532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:50867)
0619 18:56:59.407578 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:59.407631 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:59.409565 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:59.410561 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:59.410571 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:59.411313 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:56:59.412061 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.412085 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.413585 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.413590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.415711 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.415731 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.416726 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.416737 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.417008 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.417805 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:56:59.417833 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:56:59.420666 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:56:59.424554 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:59.424564 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:59.424581 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:59.425145 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:59.425628 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:56:59.425634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:56:59.425637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:56:59.425870 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:56:59.426486 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:56:59.426496 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:56:59.426998 (+ 502us) client_negotiation.cc:769] Sending connection context
0619 18:56:59.427329 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 18:56:59.428001 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":530,"threads_started":1}
I20260619 18:56:59.431036 31720 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46783
I20260619 18:56:59.431408 31720 heartbeater.cc:464] Registering TS with master...
I20260619 18:56:59.432183 31720 heartbeater.cc:511] Master 127.15.91.62:46783 requested a full tablet report, sending...
I20260619 18:56:59.433917 31263 ts_manager.cc:194] Registered new tserver with Master: 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201)
I20260619 18:56:59.435056 31263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:50867
I20260619 18:56:59.444199 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:56:59.465783 31722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.449923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.15.91.62:46783)
0619 18:56:59.450282 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:59.450291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:59.450640 (+ 349us) server_negotiation.cc:407] Connection header received
0619 18:56:59.450993 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:59.450997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:59.451077 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:59.451200 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:59.452945 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.453735 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.455446 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.455881 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.456047 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.459172 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:59.459178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:59.459181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:59.459233 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:59.462397 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:59.463435 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:59.463440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:59.463442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:59.463601 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:59.464069 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:59.464073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:59.464075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:59.464453 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:59.464737 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:59.465133 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 18:56:59.465414 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260619 18:56:59.471192 31263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:56:59.479024 31263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:56:59.508118 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.493477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:39201 (local address 127.0.0.1:47742)
0619 18:56:59.495523 (+ 2046us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:59.495587 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:59.495825 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:59.497241 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:59.497248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:59.497279 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:59.497735 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.497752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.500482 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.500491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.503731 (+ 3240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.503768 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.506549 (+ 2781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.506559 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.506938 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.507024 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 18:56:59.507224 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 18:56:59.507539 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":664,"threads_started":1}
I20260619 18:56:59.509307 31734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.493557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.15.91.3:39201)
0619 18:56:59.494552 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:59.494560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:59.496045 (+ 1485us) server_negotiation.cc:407] Connection header received
0619 18:56:59.496168 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:59.496180 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:59.496508 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:59.496800 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:59.497996 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.500226 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.504086 (+ 3860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.505832 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.506001 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.506332 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:59.508682 (+ 2350us) server_negotiation.cc:299] Negotiation successful
0619 18:56:59.508912 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":665,"threads_started":1}
I20260619 18:56:59.516176 31655 tablet_service.cc:1467] Processing CreateTablet for tablet 3268b1e89ab8432692c88b21fe57e5ac (DEFAULT_TABLE table=test-table [id=071481a492d243879e6f69ef6bc18979]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:59.518002 31655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3268b1e89ab8432692c88b21fe57e5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:59.519366 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.496078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38467 (local address 127.0.0.1:56208)
0619 18:56:59.509815 (+ 13737us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:59.509838 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:59.509955 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:59.511667 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:59.511674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:59.511715 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:59.511969 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.511984 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.514019 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.514029 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.516113 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.516137 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.518336 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.518344 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.518561 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.518598 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:56:59.518750 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 18:56:59.518951 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13582,"mutex_wait_us":35,"thread_start_us":14883,"threads_started":1}
I20260619 18:56:59.519795 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.496287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.15.91.2:38467)
0619 18:56:59.510876 (+ 14589us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:59.510887 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:59.510928 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:56:59.511059 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:59.511067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:59.511372 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:59.511667 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:59.512154 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.513741 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.516350 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.517940 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.518125 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.518352 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:59.519202 (+ 850us) server_negotiation.cc:299] Negotiation successful
0619 18:56:59.519430 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14368,"thread_start_us":14242,"threads_started":1}
I20260619 18:56:59.520661 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.495564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35204 (local address 127.15.91.1:42901)
0619 18:56:59.511084 (+ 15520us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:59.511093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:59.511128 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:56:59.511253 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:59.511261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:59.511532 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:59.511738 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:59.512662 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.513709 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.517024 (+ 3315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.518585 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.518795 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.519052 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:59.520087 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0619 18:56:59.520296 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15041,"thread_start_us":15170,"threads_started":1}
I20260619 18:56:59.521412 31735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.495173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:35204)
0619 18:56:59.495854 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:59.495873 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:59.506045 (+ 10172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:59.511982 (+ 5937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:59.511987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:59.512014 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:59.512438 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.512454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.513895 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.513903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.516792 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.516810 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.518873 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.518889 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.519117 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.519157 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:56:59.519264 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 18:56:59.519429 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":4,"thread_start_us":427,"threads_started":1}
I20260619 18:56:59.524346 31521 tablet_service.cc:1467] Processing CreateTablet for tablet 3268b1e89ab8432692c88b21fe57e5ac (DEFAULT_TABLE table=test-table [id=071481a492d243879e6f69ef6bc18979]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:59.525838 31521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3268b1e89ab8432692c88b21fe57e5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:59.526786 31387 tablet_service.cc:1467] Processing CreateTablet for tablet 3268b1e89ab8432692c88b21fe57e5ac (DEFAULT_TABLE table=test-table [id=071481a492d243879e6f69ef6bc18979]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:56:59.528762 31387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3268b1e89ab8432692c88b21fe57e5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:56:59.537577 31739 tablet_bootstrap.cc:492] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: Bootstrap starting.
I20260619 18:56:59.541662 31740 tablet_bootstrap.cc:492] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: Bootstrap starting.
I20260619 18:56:59.543346 31739 tablet_bootstrap.cc:654] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:59.544970 31739 log.cc:824] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:59.545415 31740 tablet_bootstrap.cc:654] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:59.546908 31741 tablet_bootstrap.cc:492] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: Bootstrap starting.
I20260619 18:56:59.547269 31740 log.cc:824] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:59.549441 31739 tablet_bootstrap.cc:492] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: No bootstrap required, opened a new log
I20260619 18:56:59.549861 31739 ts_tablet_manager.cc:1397] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260619 18:56:59.551326 31741 tablet_bootstrap.cc:654] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: Neither blocks nor log segments found. Creating new log.
I20260619 18:56:59.552848 31741 log.cc:824] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: Log is configured to *not* fsync() on all Append() calls
I20260619 18:56:59.556571 31740 tablet_bootstrap.cc:492] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: No bootstrap required, opened a new log
I20260619 18:56:59.556790 31741 tablet_bootstrap.cc:492] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: No bootstrap required, opened a new log
I20260619 18:56:59.557008 31740 ts_tablet_manager.cc:1397] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260619 18:56:59.557112 31741 ts_tablet_manager.cc:1397] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 18:56:59.563716 31739 raft_consensus.cc:348] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:56:59.564316 31739 raft_consensus.cc:374] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:59.564436 31739 raft_consensus.cc:729] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3868e0574b874913bb7d7a0a0953fb39, State: Initialized, Role: FOLLOWER
I20260619 18:56:59.565347 31739 consensus_queue.cc:260] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:56:59.566118 31740 raft_consensus.cc:348] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:56:59.566701 31740 raft_consensus.cc:374] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:59.566857 31740 raft_consensus.cc:729] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47795cd4d3441a0b2f064543b30c4e1, State: Initialized, Role: FOLLOWER
I20260619 18:56:59.567258 31720 heartbeater.cc:503] Master 127.15.91.62:46783 was elected leader, sending a full tablet report...
I20260619 18:56:59.567876 31739 ts_tablet_manager.cc:1428] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260619 18:56:59.568995 31740 consensus_queue.cc:260] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:56:59.569132 31741 raft_consensus.cc:348] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:56:59.569624 31741 raft_consensus.cc:374] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:56:59.569741 31741 raft_consensus.cc:729] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e68a91804f3473aa67cfacd693e4c58, State: Initialized, Role: FOLLOWER
I20260619 18:56:59.570823 31741 consensus_queue.cc:260] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:56:59.578991 31741 ts_tablet_manager.cc:1428] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: Time spent starting tablet: real 0.022s user 0.011s sys 0.008s
I20260619 18:56:59.578987 31740 ts_tablet_manager.cc:1428] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: Time spent starting tablet: real 0.022s user 0.014s sys 0.008s
I20260619 18:56:59.579636 31586 heartbeater.cc:503] Master 127.15.91.62:46783 was elected leader, sending a full tablet report...
W20260619 18:56:59.588397 31587 tablet.cc:2369] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:56:59.654413 31721 tablet.cc:2369] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:56:59.756695 31747 raft_consensus.cc:482] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:56:59.757015 31747 raft_consensus.cc:504] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
W20260619 18:56:59.758230 31453 tablet.cc:2369] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:56:59.759127 31747 leader_election.cc:288] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201), c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
I20260619 18:56:59.767287 31734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.759654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:51411 (local address 127.15.91.3:39201)
0619 18:56:59.759927 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:59.759937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:59.760413 (+ 476us) server_negotiation.cc:407] Connection header received
0619 18:56:59.760667 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:59.760675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:59.760806 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:59.760923 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:59.761856 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.763109 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.765199 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.766105 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.766303 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.766428 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:59.766818 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 18:56:59.766971 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:56:59.767481 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.760728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:45799 (local address 127.15.91.2:38467)
0619 18:56:59.760984 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:59.760991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:59.761303 (+ 312us) server_negotiation.cc:407] Connection header received
0619 18:56:59.761539 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:59.761545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:59.761617 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:59.761725 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:56:59.762303 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.763411 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.765353 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.766351 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.766555 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.766667 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:59.767050 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 18:56:59.767196 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:56:59.767513 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.759542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:39201 (local address 127.15.91.1:51411)
0619 18:56:59.760252 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:59.760270 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:59.760482 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:59.761105 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:59.761112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:59.761144 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:59.761574 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.761666 (+ 92us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.763275 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.763281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.764943 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.764964 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.766396 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.766403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.766593 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.766623 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:56:59.766788 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 18:56:59.766976 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":4,"spinlock_wait_cycles":4352,"thread_start_us":300,"threads_started":1}
I20260619 18:56:59.767665 31749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.760486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38467 (local address 127.15.91.1:45799)
0619 18:56:59.761190 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 18:56:59.761209 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:56:59.761397 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:56:59.761741 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:56:59.761745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:56:59.761769 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:56:59.762136 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.762146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.763556 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.763563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.765141 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:56:59.765156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.766652 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.766659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:56:59.766868 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.766895 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:56:59.767015 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 18:56:59.767165 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"spinlock_wait_cycles":23296,"thread_start_us":321,"threads_started":1}
I20260619 18:56:59.768868 31541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3268b1e89ab8432692c88b21fe57e5ac" candidate_uuid: "8e68a91804f3473aa67cfacd693e4c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47795cd4d3441a0b2f064543b30c4e1" is_pre_election: true
I20260619 18:56:59.769002 31675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3268b1e89ab8432692c88b21fe57e5ac" candidate_uuid: "8e68a91804f3473aa67cfacd693e4c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3868e0574b874913bb7d7a0a0953fb39" is_pre_election: true
I20260619 18:56:59.769495 31541 raft_consensus.cc:2454] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e68a91804f3473aa67cfacd693e4c58 in term 0.
I20260619 18:56:59.769568 31675 raft_consensus.cc:2454] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e68a91804f3473aa67cfacd693e4c58 in term 0.
I20260619 18:56:59.770421 31340 leader_election.cc:302] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e68a91804f3473aa67cfacd693e4c58, c47795cd4d3441a0b2f064543b30c4e1; no voters:
I20260619 18:56:59.770954 31747 raft_consensus.cc:2781] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:56:59.771134 31747 raft_consensus.cc:482] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:56:59.771283 31747 raft_consensus.cc:3037] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:59.773764 31747 raft_consensus.cc:504] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:56:59.774596 31747 leader_election.cc:288] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 1 election: Requested vote from peers 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201), c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
I20260619 18:56:59.775678 31675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3268b1e89ab8432692c88b21fe57e5ac" candidate_uuid: "8e68a91804f3473aa67cfacd693e4c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3868e0574b874913bb7d7a0a0953fb39"
I20260619 18:56:59.775797 31541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3268b1e89ab8432692c88b21fe57e5ac" candidate_uuid: "8e68a91804f3473aa67cfacd693e4c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47795cd4d3441a0b2f064543b30c4e1"
I20260619 18:56:59.775962 31675 raft_consensus.cc:3037] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:59.776104 31541 raft_consensus.cc:3037] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:56:59.778620 31541 raft_consensus.cc:2454] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e68a91804f3473aa67cfacd693e4c58 in term 1.
I20260619 18:56:59.778620 31675 raft_consensus.cc:2454] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e68a91804f3473aa67cfacd693e4c58 in term 1.
I20260619 18:56:59.779342 31340 leader_election.cc:302] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3868e0574b874913bb7d7a0a0953fb39, 8e68a91804f3473aa67cfacd693e4c58; no voters:
I20260619 18:56:59.779742 31747 raft_consensus.cc:2781] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:56:59.780189 31747 raft_consensus.cc:686] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 1 LEADER]: Becoming Leader. State: Replica: 8e68a91804f3473aa67cfacd693e4c58, State: Running, Role: LEADER
I20260619 18:56:59.780756 31747 consensus_queue.cc:237] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:56:59.786470 31263 catalog_manager.cc:5495] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e68a91804f3473aa67cfacd693e4c58 (127.15.91.1). New cstate: current_term: 1 leader_uuid: "8e68a91804f3473aa67cfacd693e4c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20260619 18:56:59.815835 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.808640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.15.91.1:42901)
0619 18:56:59.808945 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:59.808952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:59.809077 (+ 125us) server_negotiation.cc:407] Connection header received
0619 18:56:59.809383 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:59.809387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:59.809474 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:59.809657 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:56:59.810097 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.811121 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.813088 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.813560 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.813736 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.814406 (+ 670us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:56:59.815098 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:56:59.815288 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:59.815433 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 18:56:59.815533 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260619 18:56:59.826531 31541 raft_consensus.cc:1261] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1 [term 1 FOLLOWER]: Refusing update from remote peer 8e68a91804f3473aa67cfacd693e4c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:56:59.826615 31675 raft_consensus.cc:1261] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39 [term 1 FOLLOWER]: Refusing update from remote peer 8e68a91804f3473aa67cfacd693e4c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:56:59.828325 31750 consensus_queue.cc:1035] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:56:59.828869 31747 consensus_queue.cc:1035] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:56:59.846470 31753 mvcc.cc:205] Tried to move back new op lower bound from 7298643639597494272 to 7298643639426179072. Current Snapshot: MvccSnapshot[applied={T|T < 7298643639597494272}]
I20260619 18:56:59.872445 31262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:56:59.872869 31262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:56:59.877838 31262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Sending DeleteTablet for 3 replicas of tablet 3268b1e89ab8432692c88b21fe57e5ac
I20260619 18:56:59.879457 31387 tablet_service.cc:1514] Processing DeleteTablet for tablet 3268b1e89ab8432692c88b21fe57e5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:59 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:35204
I20260619 18:56:59.879447 31521 tablet_service.cc:1514] Processing DeleteTablet for tablet 3268b1e89ab8432692c88b21fe57e5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:59 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:56208
I20260619 18:56:59.879415 31655 tablet_service.cc:1514] Processing DeleteTablet for tablet 3268b1e89ab8432692c88b21fe57e5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:56:59 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:47742
I20260619 18:56:59.880924 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 31324
I20260619 18:56:59.890240 31765 tablet_replica.cc:330] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: stopping tablet replica
I20260619 18:56:59.890975 31765 raft_consensus.cc:2229] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:56:59.891258 31763 tablet_replica.cc:330] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: stopping tablet replica
I20260619 18:56:59.891407 31764 tablet_replica.cc:330] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: stopping tablet replica
I20260619 18:56:59.891618 31765 raft_consensus.cc:2258] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:59.892170 31764 raft_consensus.cc:2229] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:56:59.892616 31763 raft_consensus.cc:2229] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:56:59.893515 31764 raft_consensus.cc:2258] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:59.894186 31763 raft_consensus.cc:2258] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:56:59.894347 31765 ts_tablet_manager.cc:1905] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:59.897321 31763 ts_tablet_manager.cc:1905] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:59.899865 31764 ts_tablet_manager.cc:1905] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:56:59.903493 31765 ts_tablet_manager.cc:1918] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 19 18:56:59 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:56:59.903718 31765 log.cc:1196] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/3268b1e89ab8432692c88b21fe57e5ac
I20260619 18:56:59.904346 31765 ts_tablet_manager.cc:1939] T 3268b1e89ab8432692c88b21fe57e5ac P 3868e0574b874913bb7d7a0a0953fb39: Deleting consensus metadata
I20260619 18:56:59.906980 31247 catalog_manager.cc:4841] TS 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201): tablet 3268b1e89ab8432692c88b21fe57e5ac (table test-table [id=071481a492d243879e6f69ef6bc18979]) successfully deleted
I20260619 18:56:59.909610 31763 ts_tablet_manager.cc:1918] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:56:59.909843 31763 log.cc:1196] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/3268b1e89ab8432692c88b21fe57e5ac
I20260619 18:56:59.910478 31763 ts_tablet_manager.cc:1939] T 3268b1e89ab8432692c88b21fe57e5ac P c47795cd4d3441a0b2f064543b30c4e1: Deleting consensus metadata
I20260619 18:56:59.912732 31247 catalog_manager.cc:4841] TS c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467): tablet 3268b1e89ab8432692c88b21fe57e5ac (table test-table [id=071481a492d243879e6f69ef6bc18979]) successfully deleted
I20260619 18:56:59.915988 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:56:59.882351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.15.91.1:42901)
0619 18:56:59.882549 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 18:56:59.882556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:56:59.888964 (+ 6408us) server_negotiation.cc:407] Connection header received
0619 18:56:59.891340 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:56:59.891346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:56:59.891442 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:56:59.891525 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:56:59.893962 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.895295 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.897642 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:56:59.898443 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:56:59.898560 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:56:59.905235 (+ 6675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:56:59.905267 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:56:59.905285 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:56:59.905359 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:56:59.911324 (+ 5965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:59.913075 (+ 1751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:59.913084 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:59.913086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:59.913288 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:56:59.913796 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:56:59.913808 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:56:59.913811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:56:59.914286 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:56:59.914657 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0619 18:56:59.915189 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 18:56:59.915564 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:56:59.916857 31447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:57:00.023614 31764 ts_tablet_manager.cc:1918] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:57:00.024468 31764 log.cc:1196] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/3268b1e89ab8432692c88b21fe57e5ac
I20260619 18:57:00.025597 31764 ts_tablet_manager.cc:1939] T 3268b1e89ab8432692c88b21fe57e5ac P 8e68a91804f3473aa67cfacd693e4c58: Deleting consensus metadata
I20260619 18:57:00.025727 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 31324
W20260619 18:57:00.048816 31247 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.91.1:42901 (error 108)
I20260619 18:57:00.049234 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 31458
I20260619 18:57:00.051484 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:00.050500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:35234)
0619 18:57:00.051069 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:00.051230 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":577,"threads_started":1}
W20260619 18:57:00.052064 31247 catalog_manager.cc:4584] TS 8e68a91804f3473aa67cfacd693e4c58 (127.15.91.1:42901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3268b1e89ab8432692c88b21fe57e5ac: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Jun 19 18:57:00 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:57:00.070317 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:00.050208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.15.91.2:38467)
0619 18:57:00.050434 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:00.050441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:00.050464 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:57:00.050641 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:00.050653 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:00.050795 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:00.050928 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:00.052954 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:00.053721 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:00.055397 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:00.055845 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:00.056031 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:00.061508 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:00.061526 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:00.061544 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:00.061591 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:00.066051 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:00.067191 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:00.067201 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:00.067203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:00.067385 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:00.067876 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:00.067880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:00.067883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:00.068631 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:00.069099 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:00.069669 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 18:57:00.069981 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:57:00.071349 31581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:57:00.106093 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:00.105473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:35250)
0619 18:57:00.105761 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:00.105899 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260619 18:57:00.108230 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:00.107633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:35258)
0619 18:57:00.107881 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:00.107999 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 18:57:00.170331 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:00.168909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:35266)
0619 18:57:00.169493 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:00.170053 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 18:57:00.171461 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 31458
I20260619 18:57:00.171676 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:00.171214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:35276)
0619 18:57:00.171387 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:00.171480 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 18:57:00.189786 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 31592
Jun 19 18:57:00 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:57:00.209517 31734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:00.190615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.15.91.3:39201)
0619 18:57:00.190850 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:00.190858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:00.190926 (+ 68us) server_negotiation.cc:407] Connection header received
0619 18:57:00.191169 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:00.191175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:00.191267 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:00.191409 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:00.193687 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:00.194499 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:00.196095 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:00.196865 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:00.197053 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:00.202431 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:00.202450 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:00.202467 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:00.202513 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:00.205952 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:00.207131 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:00.207140 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:00.207143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:00.207324 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:00.207728 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:00.207732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:00.207735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:00.208154 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:00.208412 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:00.208988 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 18:57:00.209221 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:57:00.210289 31715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:57:00.253317 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:00.252718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:35284)
0619 18:57:00.253015 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:00.253135 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260619 18:57:00.254885 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:00.254241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:35294)
0619 18:57:00.254510 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:00.254597 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260619 18:57:00.308557 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 31592
I20260619 18:57:00.328894 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 31231
I20260619 18:57:00.331174 31293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:57:00.448207 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:00.447293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:35296)
0619 18:57:00.447816 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:00.447987 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328}
I20260619 18:57:00.448158 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 31231
I20260619 18:57:20.472303 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46783
--webserver_interface=127.15.91.62
--webserver_port=41145
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:57:20.848543 31773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:57:20.848894 31773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:57:20.849061 31773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:57:20.857081 31773 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:57:20.857195 31773 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:57:20.857237 31773 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:57:20.857322 31773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:57:20.857434 31773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:57:20.857554 31773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:57:20.857622 31773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:57:20.857721 31773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:57:20.868765 31773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=41145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:57:20.870316 31773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:57:20.871976 31773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:57:20.898250 31782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:20.898319 31780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:20.898620 31779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:57:20.899490 31773 server_base.cc:1034] running on GCE node
I20260619 18:57:20.900628 31773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:57:20.902629 31773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:57:20.904038 31773 hybrid_clock.cc:648] HybridClock initialized: now 1781895440903960 us; error 82 us; skew 500 ppm
I20260619 18:57:20.904596 31773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:57:20 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895440, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:57:20.913753 31773 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:57:20.915861 31773 webserver.cc:460] Webserver started at http://127.15.91.62:41145/ using document root <none> and password file <none>
I20260619 18:57:20.916821 31773 fs_manager.cc:359] Metadata directory not provided
I20260619 18:57:20.916955 31773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:57:20.924309 31773 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 18:57:20.928135 31789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:20.930181 31773 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:57:20.930406 31773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "6e0bb6a0349e44c79a22775ced98821e"
format_stamp: "Formatted at 2026-06-19 18:56:57 on dist-test-slave-dm3f"
server_key: "f8fbb66561950b6af3c4d38862092bd8"
server_key_iv: "fee797a3811fc4ce654d15563f6ad48b"
server_key_version: "encryptionkey@0"
I20260619 18:57:20.931706 31773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:57:20.946353 31773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:57:20.951738 31773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:57:20.952049 31773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:57:20.974571 31773 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:46783
I20260619 18:57:20.974606 31840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:46783 every 8 connection(s)
I20260619 18:57:20.976507 31773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:57:20.979836 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 31773
I20260619 18:57:20.980749 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:42901
--local_ip_for_outbound_sockets=127.15.91.1
--tserver_master_addrs=127.15.91.62:46783
--webserver_port=39981
--webserver_interface=127.15.91.1
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:57:20.985666 31841 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:57:20.990623 31841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Bootstrap starting.
I20260619 18:57:21.025522 31841 log.cc:824] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:57:21.042521 31841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:57:21.043366 31841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Bootstrap complete.
I20260619 18:57:21.054312 31841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:57:21.055516 31841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0bb6a0349e44c79a22775ced98821e, State: Initialized, Role: FOLLOWER
I20260619 18:57:21.056288 31841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:57:21.056648 31841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:57:21.056838 31841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:57:21.057060 31841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:57:21.061757 31841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:57:21.062503 31841 leader_election.cc:302] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e0bb6a0349e44c79a22775ced98821e; no voters:
I20260619 18:57:21.063333 31841 leader_election.cc:288] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:57:21.063481 31845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:57:21.064947 31845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 2 LEADER]: Becoming Leader. State: Replica: 6e0bb6a0349e44c79a22775ced98821e, State: Running, Role: LEADER
I20260619 18:57:21.065626 31845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:57:21.066179 31841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:57:21.068646 31847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e0bb6a0349e44c79a22775ced98821e. Latest consensus state: current_term: 2 leader_uuid: "6e0bb6a0349e44c79a22775ced98821e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } } }
I20260619 18:57:21.069192 31847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: This master's current role is: LEADER
I20260619 18:57:21.068936 31846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6e0bb6a0349e44c79a22775ced98821e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } } }
I20260619 18:57:21.069454 31846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: This master's current role is: LEADER
I20260619 18:57:21.078842 31852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:57:21.089716 31852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:57:21.092828 31852 catalog_manager.cc:1259] Loaded cluster ID: dc3411a815d14756b38280012ecaafb0
I20260619 18:57:21.093123 31852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:57:21.100104 31852 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:57:21.102610 31852 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Loaded TSK: 0
I20260619 18:57:21.136332 31852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Generated new TSK 1
I20260619 18:57:21.136854 31852 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:57:21.429373 31843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:57:21.429693 31843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:57:21.429869 31843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:57:21.438351 31843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:57:21.438508 31843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:57:21.438699 31843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:57:21.450131 31843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:42901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=39981
--tserver_master_addrs=127.15.91.62:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:57:21.451726 31843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:57:21.453292 31843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:57:21.471990 31871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:21.472270 31869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:21.474079 31868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:57:21.474090 31843 server_base.cc:1034] running on GCE node
I20260619 18:57:21.475332 31843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:57:21.476935 31843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:57:21.478308 31843 hybrid_clock.cc:648] HybridClock initialized: now 1781895441478246 us; error 50 us; skew 500 ppm
I20260619 18:57:21.478856 31843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:57:21 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895441, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:57:21.482903 31843 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:57:21.485002 31843 webserver.cc:460] Webserver started at http://127.15.91.1:39981/ using document root <none> and password file <none>
I20260619 18:57:21.485873 31843 fs_manager.cc:359] Metadata directory not provided
I20260619 18:57:21.486064 31843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:57:21.493064 31843 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:57:21.497123 31878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:21.499516 31843 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 18:57:21.499791 31843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "8e68a91804f3473aa67cfacd693e4c58"
format_stamp: "Formatted at 2026-06-19 18:56:58 on dist-test-slave-dm3f"
server_key: "d640d9397d4c3bc60c5e724a6481a7d9"
server_key_iv: "cd9144fa33a77d3b27bbce384329f986"
server_key_version: "encryptionkey@0"
I20260619 18:57:21.501048 31843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:57:21.523058 31843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:57:21.528434 31843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:57:21.528777 31843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:57:21.530016 31843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:57:21.531939 31843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:57:21.532097 31843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:21.532255 31843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:57:21.532351 31843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:21.572607 31843 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:42901
I20260619 18:57:21.572674 31990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:42901 every 8 connection(s)
I20260619 18:57:21.574334 31843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:57:21.575356 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 31843
I20260619 18:57:21.576263 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:38467
--local_ip_for_outbound_sockets=127.15.91.2
--tserver_master_addrs=127.15.91.62:46783
--webserver_port=43769
--webserver_interface=127.15.91.2
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:57:21 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895441, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:57:21.607671 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:21.579089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:55059)
0619 18:57:21.580072 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:21.580128 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:57:21.582209 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:57:21.584162 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:57:21.584172 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:57:21.584827 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:57:21.585487 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:21.585511 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:21.588450 (+ 2939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:21.588456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:21.589937 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:21.589961 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:21.591957 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:21.591964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:21.592190 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:21.593047 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:57:21.593079 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:57:21.595970 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:57:21.601990 (+ 6020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:57:21.601999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:57:21.602013 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:57:21.602588 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:57:21.603369 (+ 781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:57:21.603373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:57:21.603375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:57:21.603626 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:57:21.604906 (+ 1280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:57:21.604915 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:57:21.605442 (+ 527us) client_negotiation.cc:769] Sending connection context
0619 18:57:21.605795 (+ 353us) client_negotiation.cc:240] Negotiation successful
0619 18:57:21.606391 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":388,"threads_started":1}
I20260619 18:57:21.607934 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:21.578597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:55059 (local address 127.15.91.62:46783)
0619 18:57:21.579255 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:21.579268 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:21.580289 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 18:57:21.582997 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:21.583011 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:21.583467 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:21.583985 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:21.585777 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:21.588113 (+ 2336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:21.590205 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:21.591603 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:21.591870 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:21.596421 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:21.596437 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:21.596468 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:21.596543 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:21.601675 (+ 5132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:21.602851 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:21.602865 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:21.602876 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:21.603197 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:21.603778 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:21.603786 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:21.603791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:21.604671 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:21.604944 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:21.606109 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0619 18:57:21.606637 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":327,"threads_started":1}
I20260619 18:57:21.611263 31991 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46783
I20260619 18:57:21.611672 31991 heartbeater.cc:464] Registering TS with master...
I20260619 18:57:21.612712 31991 heartbeater.cc:511] Master 127.15.91.62:46783 requested a full tablet report, sending...
I20260619 18:57:21.615744 31806 ts_manager.cc:194] Registered new tserver with Master: 8e68a91804f3473aa67cfacd693e4c58 (127.15.91.1:42901)
I20260619 18:57:21.617398 31806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:55059
W20260619 18:57:21.949149 31995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:57:21.949492 31995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:57:21.949656 31995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:57:21.957443 31995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:57:21.957585 31995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:57:21.957801 31995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:57:21.968470 31995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:38467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=43769
--tserver_master_addrs=127.15.91.62:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:57:21.969964 31995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:57:21.971485 31995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:57:21.987066 32000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:21.987082 32001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:21.987545 32003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:57:21.988173 31995 server_base.cc:1034] running on GCE node
I20260619 18:57:21.988909 31995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:57:21.990132 31995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:57:21.991395 31995 hybrid_clock.cc:648] HybridClock initialized: now 1781895441991335 us; error 60 us; skew 500 ppm
I20260619 18:57:21.991792 31995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:57:21 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895441, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:57:21.995487 31995 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:57:21.997270 31995 webserver.cc:460] Webserver started at http://127.15.91.2:43769/ using document root <none> and password file <none>
I20260619 18:57:21.998034 31995 fs_manager.cc:359] Metadata directory not provided
I20260619 18:57:21.998167 31995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:57:22.004876 31995 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:57:22.008448 32010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:22.010221 31995 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260619 18:57:22.010416 31995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "c47795cd4d3441a0b2f064543b30c4e1"
format_stamp: "Formatted at 2026-06-19 18:56:58 on dist-test-slave-dm3f"
server_key: "741b30708213ce6a2cfe55fce83bfb09"
server_key_iv: "15034a67d35a1b549fb72c593489c025"
server_key_version: "encryptionkey@0"
I20260619 18:57:22.011611 31995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:57:22.030604 31995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:57:22.035987 31995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:57:22.036321 31995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:57:22.037516 31995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:57:22.039176 31995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:57:22.039304 31995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:22.039391 31995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:57:22.039455 31995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:22.079387 31995 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:38467
I20260619 18:57:22.079449 32122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:38467 every 8 connection(s)
I20260619 18:57:22.081226 31995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:57:22.086942 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 31995
I20260619 18:57:22.087831 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:39201
--local_ip_for_outbound_sockets=127.15.91.3
--tserver_master_addrs=127.15.91.62:46783
--webserver_port=43849
--webserver_interface=127.15.91.3
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:57:22 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895441, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:57:22.105446 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.084291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:50753 (local address 127.15.91.62:46783)
0619 18:57:22.084519 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.084524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.085911 (+ 1387us) server_negotiation.cc:407] Connection header received
0619 18:57:22.088315 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.088321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.088451 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.088586 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:22.090435 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.091284 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.093092 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.093581 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.093739 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.098260 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:22.098266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:22.098269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:22.098309 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:22.101555 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:22.102573 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:22.102579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:22.102581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:22.102808 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:22.103338 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:22.103343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:22.103347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:22.103707 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:22.103891 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:22.104749 (+ 858us) server_negotiation.cc:299] Negotiation successful
0619 18:57:22.105050 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 18:57:22.106786 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.084788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:50753)
0619 18:57:22.085629 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:22.085750 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0619 18:57:22.087839 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:57:22.088890 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:57:22.088900 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:57:22.089512 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:57:22.090168 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.090188 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.091499 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.091504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.092867 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.092884 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.093865 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.093873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.094152 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.095001 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:57:22.095027 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:57:22.097899 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:57:22.101794 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:57:22.101806 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:57:22.101831 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:57:22.102369 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:57:22.102947 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:57:22.102952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:57:22.102956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:57:22.103200 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:57:22.103894 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:57:22.103905 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:57:22.104329 (+ 424us) client_negotiation.cc:769] Sending connection context
0619 18:57:22.104644 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 18:57:22.105338 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":335,"threads_started":1}
I20260619 18:57:22.108996 32123 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46783
I20260619 18:57:22.109427 32123 heartbeater.cc:464] Registering TS with master...
I20260619 18:57:22.110275 32123 heartbeater.cc:511] Master 127.15.91.62:46783 requested a full tablet report, sending...
I20260619 18:57:22.112233 31806 ts_manager.cc:194] Registered new tserver with Master: c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
I20260619 18:57:22.113108 31806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:50753
W20260619 18:57:22.462050 32126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:57:22.462389 32126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:57:22.462579 32126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:57:22.470672 32126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:57:22.470850 32126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:57:22.471073 32126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:57:22.482018 32126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:39201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=43849
--tserver_master_addrs=127.15.91.62:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:57:22.483611 32126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:57:22.485100 32126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:57:22.500878 32132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:22.500804 32131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:22.500772 32134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:57:22.501991 32126 server_base.cc:1034] running on GCE node
I20260619 18:57:22.502789 32126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:57:22.503933 32126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:57:22.505209 32126 hybrid_clock.cc:648] HybridClock initialized: now 1781895442505135 us; error 59 us; skew 500 ppm
I20260619 18:57:22.505602 32126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:57:22 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895442, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:57:22.509553 32126 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:57:22.511489 32126 webserver.cc:460] Webserver started at http://127.15.91.3:43849/ using document root <none> and password file <none>
I20260619 18:57:22.512341 32126 fs_manager.cc:359] Metadata directory not provided
I20260619 18:57:22.512531 32126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:57:22.519246 32126 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:57:22.522630 32141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:22.524641 32126 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:57:22.524911 32126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "3868e0574b874913bb7d7a0a0953fb39"
format_stamp: "Formatted at 2026-06-19 18:56:59 on dist-test-slave-dm3f"
server_key: "cd60e9c3febb477df37c7aff3417c503"
server_key_iv: "6548cc574ecc14e3dce9b5b0640b3926"
server_key_version: "encryptionkey@0"
I20260619 18:57:22.526041 32126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:57:22.547629 32126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:57:22.553311 32126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:57:22.553674 32126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:57:22.554881 32126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:57:22.556674 32126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:57:22.556872 32126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:22.557041 32126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:57:22.557140 32126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:22.595299 32126 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:39201
I20260619 18:57:22.595359 32253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:39201 every 8 connection(s)
I20260619 18:57:22.597133 32126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:57:22.605780 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 32126
Jun 19 18:57:22 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895442, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:57:22.621721 31991 heartbeater.cc:503] Master 127.15.91.62:46783 was elected leader, sending a full tablet report...
I20260619 18:57:22.625880 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.600335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:41139 (local address 127.15.91.62:46783)
0619 18:57:22.600586 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.600593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.602222 (+ 1629us) server_negotiation.cc:407] Connection header received
0619 18:57:22.604364 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.604369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.604460 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.604567 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:22.607169 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.608451 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.611364 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.612140 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.612291 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.617528 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:22.617534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:22.617537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:22.617577 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:22.621658 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:22.622807 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:22.622812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:22.622815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:22.623018 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:22.623703 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:22.623709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:22.623712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:22.624138 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:22.624283 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:22.625183 (+ 900us) server_negotiation.cc:299] Negotiation successful
0619 18:57:22.625454 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":94}
I20260619 18:57:22.626672 32257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.607058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.15.91.62:46783)
0619 18:57:22.608395 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.608402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.608422 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:57:22.608488 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.608493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.608582 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.608682 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:22.611130 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.612319 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.614468 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.615065 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.615182 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.618927 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:22.618934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:22.618940 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:22.619032 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:22.622808 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:22.624325 (+ 1517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:22.624328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:22.624330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:22.624452 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:22.624895 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:22.624901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:22.624904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:22.625190 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:22.625497 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:22.625850 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 18:57:22.626114 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":1208,"thread_start_us":1072,"threads_started":1}
I20260619 18:57:22.627327 32256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.601083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:41139)
0619 18:57:22.602010 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:22.602059 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:57:22.603980 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:57:22.604861 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:57:22.604872 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:57:22.605832 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:57:22.606917 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.606956 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.608733 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.608739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.611197 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.611218 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.612500 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.612509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.612913 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.613813 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:57:22.613857 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:57:22.617168 (+ 3311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:57:22.621901 (+ 4733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:57:22.621913 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:57:22.621960 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:57:22.622590 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:57:22.623178 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:57:22.623184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:57:22.623187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:57:22.623534 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:57:22.624305 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:57:22.624313 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:57:22.624721 (+ 408us) client_negotiation.cc:769] Sending connection context
0619 18:57:22.625056 (+ 335us) client_negotiation.cc:240] Negotiation successful
0619 18:57:22.625684 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":361,"threads_started":1}
I20260619 18:57:22.628968 32254 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46783
I20260619 18:57:22.629407 32254 heartbeater.cc:464] Registering TS with master...
I20260619 18:57:22.630440 32254 heartbeater.cc:511] Master 127.15.91.62:46783 requested a full tablet report, sending...
I20260619 18:57:22.632335 31806 ts_manager.cc:194] Registered new tserver with Master: 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201)
I20260619 18:57:22.633294 31806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:41139
I20260619 18:57:22.642841 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:57:22.650967 32257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.644087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.15.91.62:46783)
0619 18:57:22.644406 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.644413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.644712 (+ 299us) server_negotiation.cc:407] Connection header received
0619 18:57:22.645039 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.645044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.645212 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.645410 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:57:22.646002 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.646706 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.648522 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.649013 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.649185 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.649849 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:57:22.650121 (+ 272us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 18:57:22.650588 (+ 467us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48812: token expired
Metrics: {"server-negotiator.queue_time_us":102}
W20260619 18:57:22.651145 32257 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48812: token expired
W20260619 18:57:22.651649 31726 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.91.62:46783)
I20260619 18:57:22.668069 32257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.652421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.15.91.62:46783)
0619 18:57:22.652613 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.652619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.652637 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:57:22.652691 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.652694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.652760 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.652876 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:22.654470 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.655775 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.657386 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.657858 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.657995 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.661517 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:22.661523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:22.661527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:22.661575 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:22.664780 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:22.665666 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:22.665672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:22.665674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:22.665845 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:22.666416 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:22.666419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:22.666421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:22.666920 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:22.667131 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:22.667506 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 18:57:22.667719 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:57:22.672271 31806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:57:22.681100 31806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:57:22.707180 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.695765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.15.91.1:42901)
0619 18:57:22.696409 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.696417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.696541 (+ 124us) server_negotiation.cc:407] Connection header received
0619 18:57:22.696836 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.696842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.697177 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.698431 (+ 1254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:57:22.698768 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.700797 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.704013 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.705621 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.705799 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.706076 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:22.706570 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 18:57:22.706799 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":1073,"threads_started":1}
I20260619 18:57:22.707288 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.695252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:46384)
0619 18:57:22.696384 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:22.696426 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:57:22.696631 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:57:22.697467 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:57:22.697479 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:57:22.697530 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:57:22.698120 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.698142 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.700962 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.700971 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.703768 (+ 2797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.703785 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.705885 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.705896 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.706268 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.706337 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 18:57:22.706501 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 18:57:22.706820 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":2,"thread_start_us":742,"threads_started":1}
I20260619 18:57:22.708181 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.696990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:39201 (local address 127.0.0.1:57504)
0619 18:57:22.697608 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:22.697633 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:57:22.697754 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:57:22.699741 (+ 1987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:57:22.699746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:57:22.699773 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:57:22.700196 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.700211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.702380 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.702385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.704654 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.704671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.706933 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.706941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.707146 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.707183 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:57:22.707261 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:57:22.707761 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":321,"threads_started":1}
I20260619 18:57:22.709867 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.697685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38467 (local address 127.0.0.1:39736)
0619 18:57:22.698333 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:22.698348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:57:22.698440 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:57:22.701427 (+ 2987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:57:22.701436 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:57:22.701467 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:57:22.701972 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.701985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.703767 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.703772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.705992 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.706011 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.708945 (+ 2934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.708951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.709134 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.709169 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:57:22.709286 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 18:57:22.709444 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":59,"thread_start_us":400,"threads_started":1}
I20260619 18:57:22.709858 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.697288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.15.91.3:39201)
0619 18:57:22.699012 (+ 1724us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.699023 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.699065 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:57:22.699171 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.699182 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.699510 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.700446 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:57:22.700789 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.702206 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.704880 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.706628 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.708828 (+ 2200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.709226 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:22.709312 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 18:57:22.709516 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1539,"thread_start_us":1313,"threads_started":1}
I20260619 18:57:22.709928 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.697920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.15.91.2:38467)
0619 18:57:22.700698 (+ 2778us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.700708 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.700740 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:57:22.700844 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.700856 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.701192 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.701414 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:57:22.702174 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.703585 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.706229 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.708266 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.708542 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.708753 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:22.709377 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 18:57:22.709571 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2601,"thread_start_us":2474,"threads_started":1}
I20260619 18:57:22.711951 31926 tablet_service.cc:1467] Processing CreateTablet for tablet d585c27d51cd4b018bb1111de519d3ed (DEFAULT_TABLE table=test-table [id=3d5757b9caeb4c738f1d6481d1390414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:57:22.713579 31926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d585c27d51cd4b018bb1111de519d3ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:57:22.713964 32058 tablet_service.cc:1467] Processing CreateTablet for tablet d585c27d51cd4b018bb1111de519d3ed (DEFAULT_TABLE table=test-table [id=3d5757b9caeb4c738f1d6481d1390414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:57:22.715636 32058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d585c27d51cd4b018bb1111de519d3ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:57:22.715795 32189 tablet_service.cc:1467] Processing CreateTablet for tablet d585c27d51cd4b018bb1111de519d3ed (DEFAULT_TABLE table=test-table [id=3d5757b9caeb4c738f1d6481d1390414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:57:22.717433 32189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d585c27d51cd4b018bb1111de519d3ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:57:22.729707 32270 tablet_bootstrap.cc:492] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Bootstrap starting.
I20260619 18:57:22.733489 32270 tablet_bootstrap.cc:654] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Neither blocks nor log segments found. Creating new log.
I20260619 18:57:22.734776 32272 tablet_bootstrap.cc:492] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: Bootstrap starting.
I20260619 18:57:22.735144 32270 log.cc:824] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Log is configured to *not* fsync() on all Append() calls
I20260619 18:57:22.736914 32271 tablet_bootstrap.cc:492] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: Bootstrap starting.
I20260619 18:57:22.740170 32272 tablet_bootstrap.cc:654] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: Neither blocks nor log segments found. Creating new log.
I20260619 18:57:22.740330 32271 tablet_bootstrap.cc:654] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: Neither blocks nor log segments found. Creating new log.
I20260619 18:57:22.741667 32272 log.cc:824] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: Log is configured to *not* fsync() on all Append() calls
I20260619 18:57:22.741736 32271 log.cc:824] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:57:22.744091 32270 tablet_bootstrap.cc:492] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: No bootstrap required, opened a new log
I20260619 18:57:22.744462 32270 ts_tablet_manager.cc:1397] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260619 18:57:22.745157 32272 tablet_bootstrap.cc:492] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: No bootstrap required, opened a new log
I20260619 18:57:22.745270 32271 tablet_bootstrap.cc:492] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: No bootstrap required, opened a new log
I20260619 18:57:22.745489 32272 ts_tablet_manager.cc:1397] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260619 18:57:22.745559 32271 ts_tablet_manager.cc:1397] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 18:57:22.754164 32272 raft_consensus.cc:348] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:22.754671 32272 raft_consensus.cc:374] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:57:22.754825 32272 raft_consensus.cc:729] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3868e0574b874913bb7d7a0a0953fb39, State: Initialized, Role: FOLLOWER
I20260619 18:57:22.755592 32272 consensus_queue.cc:260] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:22.756898 32254 heartbeater.cc:503] Master 127.15.91.62:46783 was elected leader, sending a full tablet report...
I20260619 18:57:22.756678 32270 raft_consensus.cc:348] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:22.757252 32270 raft_consensus.cc:374] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:57:22.757347 32270 raft_consensus.cc:729] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e68a91804f3473aa67cfacd693e4c58, State: Initialized, Role: FOLLOWER
I20260619 18:57:22.757062 32271 raft_consensus.cc:348] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:22.757561 32271 raft_consensus.cc:374] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:57:22.757663 32271 raft_consensus.cc:729] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47795cd4d3441a0b2f064543b30c4e1, State: Initialized, Role: FOLLOWER
I20260619 18:57:22.757745 32272 ts_tablet_manager.cc:1428] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: Time spent starting tablet: real 0.012s user 0.003s sys 0.010s
I20260619 18:57:22.758188 32270 consensus_queue.cc:260] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:22.758315 32271 consensus_queue.cc:260] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:22.763175 32271 ts_tablet_manager.cc:1428] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260619 18:57:22.763280 32123 heartbeater.cc:503] Master 127.15.91.62:46783 was elected leader, sending a full tablet report...
I20260619 18:57:22.764786 32270 ts_tablet_manager.cc:1428] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260619 18:57:22.765854 32277 raft_consensus.cc:482] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:57:22.766206 32277 raft_consensus.cc:504] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:22.768606 32277 leader_election.cc:288] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201), c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
I20260619 18:57:22.775928 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.769271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:59107 (local address 127.15.91.3:39201)
0619 18:57:22.769542 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.769549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.770116 (+ 567us) server_negotiation.cc:407] Connection header received
0619 18:57:22.770201 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.770207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.770286 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.770415 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:57:22.771033 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.771786 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.773853 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.774906 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.775064 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.775182 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:22.775518 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 18:57:22.775657 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 18:57:22.776113 32279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.768842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:39201 (local address 127.15.91.1:59107)
0619 18:57:22.769653 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:22.769670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:57:22.769842 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:57:22.770433 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:57:22.770437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:57:22.770472 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:57:22.770887 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.770897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.771946 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.771951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.773664 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.773679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.775136 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.775142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.775325 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.775351 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:57:22.775486 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 18:57:22.775645 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":29,"thread_start_us":441,"threads_started":1}
I20260619 18:57:22.776239 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.769987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:37889 (local address 127.15.91.2:38467)
0619 18:57:22.770192 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.770197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.770467 (+ 270us) server_negotiation.cc:407] Connection header received
0619 18:57:22.770672 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.770678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.770775 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.770884 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:57:22.771399 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.772154 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.774193 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.775222 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.775393 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.775498 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:22.775816 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 18:57:22.775962 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:57:22.776407 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.769744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38467 (local address 127.15.91.1:37889)
0619 18:57:22.770354 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:22.770369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:57:22.770535 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:57:22.770898 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:57:22.770903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:57:22.770925 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:57:22.771255 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.771263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.772286 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.772293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.773981 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:22.773996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.775472 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.775478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:22.775651 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.775675 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:57:22.775792 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 18:57:22.775928 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":17,"thread_start_us":336,"threads_started":1}
I20260619 18:57:22.777604 32209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d585c27d51cd4b018bb1111de519d3ed" candidate_uuid: "8e68a91804f3473aa67cfacd693e4c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3868e0574b874913bb7d7a0a0953fb39" is_pre_election: true
I20260619 18:57:22.777652 32078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d585c27d51cd4b018bb1111de519d3ed" candidate_uuid: "8e68a91804f3473aa67cfacd693e4c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47795cd4d3441a0b2f064543b30c4e1" is_pre_election: true
I20260619 18:57:22.778151 32209 raft_consensus.cc:2454] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e68a91804f3473aa67cfacd693e4c58 in term 0.
I20260619 18:57:22.778151 32078 raft_consensus.cc:2454] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e68a91804f3473aa67cfacd693e4c58 in term 0.
I20260619 18:57:22.778927 31879 leader_election.cc:302] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e68a91804f3473aa67cfacd693e4c58, c47795cd4d3441a0b2f064543b30c4e1; no voters:
I20260619 18:57:22.779392 32277 raft_consensus.cc:2781] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:57:22.779583 32277 raft_consensus.cc:482] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:57:22.779702 32277 raft_consensus.cc:3037] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:57:22.783923 32277 raft_consensus.cc:504] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:22.784746 32277 leader_election.cc:288] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 1 election: Requested vote from peers 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201), c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
I20260619 18:57:22.785382 32209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d585c27d51cd4b018bb1111de519d3ed" candidate_uuid: "8e68a91804f3473aa67cfacd693e4c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3868e0574b874913bb7d7a0a0953fb39"
I20260619 18:57:22.785442 32078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d585c27d51cd4b018bb1111de519d3ed" candidate_uuid: "8e68a91804f3473aa67cfacd693e4c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47795cd4d3441a0b2f064543b30c4e1"
I20260619 18:57:22.785647 32209 raft_consensus.cc:3037] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:57:22.785668 32078 raft_consensus.cc:3037] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:57:22.788293 32209 raft_consensus.cc:2454] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e68a91804f3473aa67cfacd693e4c58 in term 1.
I20260619 18:57:22.788293 32078 raft_consensus.cc:2454] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e68a91804f3473aa67cfacd693e4c58 in term 1.
I20260619 18:57:22.788975 31879 leader_election.cc:302] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e68a91804f3473aa67cfacd693e4c58, c47795cd4d3441a0b2f064543b30c4e1; no voters:
I20260619 18:57:22.789386 32277 raft_consensus.cc:2781] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:57:22.790036 32277 raft_consensus.cc:686] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 LEADER]: Becoming Leader. State: Replica: 8e68a91804f3473aa67cfacd693e4c58, State: Running, Role: LEADER
I20260619 18:57:22.790632 32277 consensus_queue.cc:237] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:22.796634 31806 catalog_manager.cc:5495] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e68a91804f3473aa67cfacd693e4c58 (127.15.91.1). New cstate: current_term: 1 leader_uuid: "8e68a91804f3473aa67cfacd693e4c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } health_report { overall_health: UNKNOWN } } }
W20260619 18:57:22.827812 31992 tablet.cc:2369] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:57:22.833494 32124 tablet.cc:2369] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:57:22.838882 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.832560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.15.91.1:42901)
0619 18:57:22.832803 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.832811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.832871 (+ 60us) server_negotiation.cc:407] Connection header received
0619 18:57:22.833195 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.833199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.833286 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.833497 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:57:22.833933 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.834698 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.836382 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.836860 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.837034 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.837658 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:57:22.838200 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:57:22.838376 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:22.838474 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 18:57:22.838570 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:57:22.848842 32209 raft_consensus.cc:1261] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39 [term 1 FOLLOWER]: Refusing update from remote peer 8e68a91804f3473aa67cfacd693e4c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:57:22.848979 32078 raft_consensus.cc:1261] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1 [term 1 FOLLOWER]: Refusing update from remote peer 8e68a91804f3473aa67cfacd693e4c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 18:57:22.849135 32255 tablet.cc:2369] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:57:22.850142 32277 consensus_queue.cc:1035] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:57:22.850544 32281 consensus_queue.cc:1035] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:57:22.875110 32284 mvcc.cc:205] Tried to move back new op lower bound from 7298643733895168000 to 7298643733675667456. Current Snapshot: MvccSnapshot[applied={T|T < 7298643733895168000}]
I20260619 18:57:22.892298 31805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:57:22.892671 31805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:57:22.897614 31805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Sending DeleteTablet for 3 replicas of tablet d585c27d51cd4b018bb1111de519d3ed
I20260619 18:57:22.899309 31926 tablet_service.cc:1514] Processing DeleteTablet for tablet d585c27d51cd4b018bb1111de519d3ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:57:22 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:46384
I20260619 18:57:22.899350 32189 tablet_service.cc:1514] Processing DeleteTablet for tablet d585c27d51cd4b018bb1111de519d3ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:57:22 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:57504
I20260619 18:57:22.899377 32058 tablet_service.cc:1514] Processing DeleteTablet for tablet d585c27d51cd4b018bb1111de519d3ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:57:22 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:39736
I20260619 18:57:22.900733 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 31843
I20260619 18:57:22.935225 32295 tablet_replica.cc:330] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: stopping tablet replica
I20260619 18:57:22.935643 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:22.902168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46398 (local address 127.15.91.1:42901)
0619 18:57:22.902370 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:22.902379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:22.902410 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:57:22.902485 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:22.902490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:22.902593 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:22.902738 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:22.905182 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.906599 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.909123 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:22.909956 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:22.910119 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:22.919355 (+ 9236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:22.919371 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:22.919392 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:22.919455 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:22.924312 (+ 4857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:22.925553 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:22.925565 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:22.925569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:22.925798 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:22.927379 (+ 1581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:22.927386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:22.927389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:22.927959 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:22.928349 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:22.929797 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0619 18:57:22.935201 (+ 5404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 18:57:22.935985 32295 raft_consensus.cc:2229] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:57:22.936676 32295 raft_consensus.cc:2258] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:57:22.936928 31986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:57:22.937698 32293 tablet_replica.cc:330] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: stopping tablet replica
I20260619 18:57:22.941711 32295 ts_tablet_manager.cc:1905] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:57:22.941619 32293 raft_consensus.cc:2229] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:57:22.942507 32293 raft_consensus.cc:2258] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:57:22.945236 32293 ts_tablet_manager.cc:1905] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:57:22.949536 32295 ts_tablet_manager.cc:1918] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:57:22.949765 32295 log.cc:1196] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/d585c27d51cd4b018bb1111de519d3ed
I20260619 18:57:22.950376 32295 ts_tablet_manager.cc:1939] T d585c27d51cd4b018bb1111de519d3ed P c47795cd4d3441a0b2f064543b30c4e1: Deleting consensus metadata
I20260619 18:57:22.951990 32293 ts_tablet_manager.cc:1918] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:57:22.952251 32293 log.cc:1196] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/d585c27d51cd4b018bb1111de519d3ed
I20260619 18:57:22.952759 31790 catalog_manager.cc:4841] TS c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467): tablet d585c27d51cd4b018bb1111de519d3ed (table test-table [id=3d5757b9caeb4c738f1d6481d1390414]) successfully deleted
I20260619 18:57:22.952921 32293 ts_tablet_manager.cc:1939] T d585c27d51cd4b018bb1111de519d3ed P 3868e0574b874913bb7d7a0a0953fb39: Deleting consensus metadata
I20260619 18:57:22.954880 31790 catalog_manager.cc:4841] TS 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201): tablet d585c27d51cd4b018bb1111de519d3ed (table test-table [id=3d5757b9caeb4c738f1d6481d1390414]) successfully deleted
I20260619 18:57:23.052639 32294 tablet_replica.cc:330] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: stopping tablet replica
I20260619 18:57:23.053021 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 31843
W20260619 18:57:23.074005 31790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.91.1:42901 (error 108)
I20260619 18:57:23.074235 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 31995
I20260619 18:57:23.076440 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:23.075993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:46408)
0619 18:57:23.076171 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:23.076295 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 18:57:23.076728 31790 catalog_manager.cc:4584] TS 8e68a91804f3473aa67cfacd693e4c58 (127.15.91.1:42901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d585c27d51cd4b018bb1111de519d3ed: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
I20260619 18:57:23.091239 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:23.075428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.15.91.2:38467)
0619 18:57:23.075615 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:23.075620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:23.075638 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:57:23.075698 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:23.075702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:23.075769 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:23.075919 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:23.077525 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:23.078300 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:23.080025 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:23.080522 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:23.080719 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:23.083916 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:23.083933 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:23.083949 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:23.083997 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:23.087664 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:23.088625 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:23.088634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:23.088637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:23.088794 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:23.089272 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:23.089279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:23.089287 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:23.089776 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:23.090125 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:23.090539 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 18:57:23.090848 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:57:23.092258 32118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:57:23.130698 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:23.130149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:46424)
0619 18:57:23.130356 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:23.130541 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 18:57:23.131955 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:23.131522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:46430)
0619 18:57:23.131691 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:23.131791 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 18:57:23.186966 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 31995
I20260619 18:57:23.193071 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:23.192342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:46442)
0619 18:57:23.192662 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:23.192800 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260619 18:57:23.194481 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:23.193926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:46456)
0619 18:57:23.194157 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:23.194274 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 18:57:23.205838 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 32126
I20260619 18:57:23.221302 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:23.206871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.15.91.3:39201)
0619 18:57:23.207096 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:23.207102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:23.207127 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:57:23.207312 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:23.207317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:23.207395 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:23.207514 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:23.209013 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:23.209776 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:23.211435 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:23.211851 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:23.212016 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:23.214933 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:23.214951 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:23.214968 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:23.215031 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:23.217916 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:23.218899 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:23.218907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:23.218909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:23.219082 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:23.219553 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:23.219558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:23.219560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:23.219956 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:23.220303 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:23.220654 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 18:57:23.220931 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:57:23.222123 32249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:57:23.275513 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:23.274874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:46468)
0619 18:57:23.275114 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:23.275270 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 18:57:23.276808 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:23.276346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:46478)
0619 18:57:23.276566 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:23.276652 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 18:57:23.316289 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 32126
I20260619 18:57:23.334338 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 31773
I20260619 18:57:23.335758 31836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:57:23.462386 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:23.461204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:42901 (local address 127.0.0.1:46480)
0619 18:57:23.461938 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:23.462077 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578}
I20260619 18:57:23.463148 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 31773
I20260619 18:57:43.484395 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46783
--webserver_interface=127.15.91.62
--webserver_port=41145
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:57:43.871630 32301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:57:43.871975 32301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:57:43.872141 32301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:57:43.880347 32301 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:57:43.880475 32301 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:57:43.880518 32301 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:57:43.880604 32301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:57:43.880702 32301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:57:43.880816 32301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:57:43.880875 32301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:57:43.880926 32301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:57:43.891968 32301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=41145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:57:43.893496 32301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:57:43.895156 32301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:57:43.915242 32306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:43.915576 32307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:43.916340 32309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:57:43.916917 32301 server_base.cc:1034] running on GCE node
I20260619 18:57:43.918234 32301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:57:43.920601 32301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:57:43.922065 32301 hybrid_clock.cc:648] HybridClock initialized: now 1781895463921863 us; error 104 us; skew 500 ppm
I20260619 18:57:43.922930 32301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:57:43 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895463, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:57:43.927289 32301 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:57:43.929225 32301 webserver.cc:460] Webserver started at http://127.15.91.62:41145/ using document root <none> and password file <none>
I20260619 18:57:43.930130 32301 fs_manager.cc:359] Metadata directory not provided
I20260619 18:57:43.930253 32301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:57:43.937506 32301 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 18:57:43.941779 32316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:43.943930 32301 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:57:43.944209 32301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "6e0bb6a0349e44c79a22775ced98821e"
format_stamp: "Formatted at 2026-06-19 18:56:57 on dist-test-slave-dm3f"
server_key: "f8fbb66561950b6af3c4d38862092bd8"
server_key_iv: "fee797a3811fc4ce654d15563f6ad48b"
server_key_version: "encryptionkey@0"
I20260619 18:57:43.945406 32301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:57:43.991940 32301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:57:43.997115 32301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:57:43.997483 32301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:57:44.020645 32301 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:46783
I20260619 18:57:44.020685 32367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:46783 every 8 connection(s)
I20260619 18:57:44.022461 32301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:57:44.024235 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 32301
I20260619 18:57:44.025687 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:42901
--local_ip_for_outbound_sockets=127.15.91.1
--tserver_master_addrs=127.15.91.62:46783
--webserver_port=39981
--webserver_interface=127.15.91.1
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:57:44.035642 32368 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:57:44.039631 32368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Bootstrap starting.
I20260619 18:57:44.063437 32368 log.cc:824] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:57:44.082352 32368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:57:44.083092 32368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Bootstrap complete.
I20260619 18:57:44.094122 32368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:57:44.095321 32368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0bb6a0349e44c79a22775ced98821e, State: Initialized, Role: FOLLOWER
I20260619 18:57:44.096200 32368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:57:44.096509 32368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:57:44.096642 32368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:57:44.096855 32368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:57:44.101341 32368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:57:44.102015 32368 leader_election.cc:302] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e0bb6a0349e44c79a22775ced98821e; no voters:
I20260619 18:57:44.102813 32368 leader_election.cc:288] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 18:57:44.103016 32372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 3 FOLLOWER]: Leader election won for term 3
I20260619 18:57:44.105028 32372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [term 3 LEADER]: Becoming Leader. State: Replica: 6e0bb6a0349e44c79a22775ced98821e, State: Running, Role: LEADER
I20260619 18:57:44.105702 32368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:57:44.105836 32372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } }
I20260619 18:57:44.109272 32374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e0bb6a0349e44c79a22775ced98821e. Latest consensus state: current_term: 3 leader_uuid: "6e0bb6a0349e44c79a22775ced98821e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } } }
I20260619 18:57:44.109844 32374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: This master's current role is: LEADER
I20260619 18:57:44.110560 32373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6e0bb6a0349e44c79a22775ced98821e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0bb6a0349e44c79a22775ced98821e" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46783 } } }
I20260619 18:57:44.110980 32373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e [sys.catalog]: This master's current role is: LEADER
I20260619 18:57:44.121485 32380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:57:44.135834 32380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:57:44.138314 32380 catalog_manager.cc:1259] Loaded cluster ID: dc3411a815d14756b38280012ecaafb0
I20260619 18:57:44.138612 32380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:57:44.145078 32380 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:57:44.148176 32380 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Loaded TSK: 0
I20260619 18:57:44.148595 32380 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6e0bb6a0349e44c79a22775ced98821e: Loaded TSK: 1
F20260619 18:57:44.150650 32380 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4d6a8be9ad google::LogMessage::Fail() at ??:0
I20260619 18:57:44.156682 32390 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f4d6a8c188a google::LogMessage::SendToLog() at ??:0
@ 0x7f4d6a8be4de google::LogMessage::Flush() at ??:0
@ 0x7f4d6a8bfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4d6e6227d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4d6e61173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4d802ed1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4d802ec783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4d803782a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f4d803780c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f4d803c25e9 std::function<>::operator()() at ??:0
@ 0x7f4d802f0cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4d802eec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4d80376610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f4d80376439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4d6d50a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4d6d50d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4d6d50d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4d6d4e8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4d707c26db start_thread at ??:0
@ 0x7f4d6975861f clone at ??:0
W20260619 18:57:44.471794 32370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:57:44.472180 32370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:57:44.472352 32370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:57:44.480906 32370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:57:44.481112 32370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:57:44.481302 32370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:57:44.492871 32370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:42901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=39981
--tserver_master_addrs=127.15.91.62:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:57:44.494417 32370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:57:44.496132 32370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:57:44.511545 32417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:44.511607 32416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:44.511626 32419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:57:44.512554 32370 server_base.cc:1034] running on GCE node
I20260619 18:57:44.513361 32370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:57:44.514530 32370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:57:44.515754 32370 hybrid_clock.cc:648] HybridClock initialized: now 1781895464515692 us; error 61 us; skew 500 ppm
I20260619 18:57:44.516201 32370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:57:44 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895464, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:57:44.525300 32370 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:57:44.527444 32370 webserver.cc:460] Webserver started at http://127.15.91.1:39981/ using document root <none> and password file <none>
I20260619 18:57:44.528270 32370 fs_manager.cc:359] Metadata directory not provided
I20260619 18:57:44.528416 32370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:57:44.535095 32370 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:57:44.538920 32426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:44.540706 32370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:57:44.540980 32370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "8e68a91804f3473aa67cfacd693e4c58"
format_stamp: "Formatted at 2026-06-19 18:56:58 on dist-test-slave-dm3f"
server_key: "d640d9397d4c3bc60c5e724a6481a7d9"
server_key_iv: "cd9144fa33a77d3b27bbce384329f986"
server_key_version: "encryptionkey@0"
I20260619 18:57:44.542172 32370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:57:44.559751 32370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:57:44.565140 32370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:57:44.565495 32370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:57:44.566785 32370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:57:44.568945 32433 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 18:57:44.575609 32370 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:57:44.575834 32370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260619 18:57:44.575991 32370 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:57:44.579931 32370 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:57:44.580094 32370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.001s
I20260619 18:57:44.580592 32433 tablet_bootstrap.cc:492] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Bootstrap starting.
I20260619 18:57:44.613101 32433 log.cc:824] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Log is configured to *not* fsync() on all Append() calls
I20260619 18:57:44.629822 32433 tablet_bootstrap.cc:492] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:57:44.630582 32433 tablet_bootstrap.cc:492] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Bootstrap complete.
I20260619 18:57:44.631242 32370 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:42901
I20260619 18:57:44.631330 32540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:42901 every 8 connection(s)
I20260619 18:57:44.632230 32433 ts_tablet_manager.cc:1397] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Time spent bootstrapping tablet: real 0.052s user 0.045s sys 0.004s
I20260619 18:57:44.632896 32370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:57:44.638639 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 32370
I20260619 18:57:44.639519 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:38467
--local_ip_for_outbound_sockets=127.15.91.2
--tserver_master_addrs=127.15.91.62:46783
--webserver_port=43769
--webserver_interface=127.15.91.2
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:57:44.640324 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:44.636846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:36709)
0619 18:57:44.638139 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:44.638590 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":518,"threads_started":1}
W20260619 18:57:44.641459 32541 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:46783 (0 consecutive failures): Network error: Failed to ping master at 127.15.91.62:46783: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
I20260619 18:57:44.642928 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:44.642288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:55081)
0619 18:57:44.642567 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:44.642683 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 18:57:44.642848 32433 raft_consensus.cc:348] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:44.644359 32433 raft_consensus.cc:729] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e68a91804f3473aa67cfacd693e4c58, State: Initialized, Role: FOLLOWER
I20260619 18:57:44.644448 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:44.643847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:33929)
0619 18:57:44.644078 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:44.644204 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 18:57:44.644829 32541 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:57:44.645231 32433 consensus_queue.cc:260] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:44.647320 32433 ts_tablet_manager.cc:1428] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260619 18:57:44.648978 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:44.648511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:60003)
0619 18:57:44.648721 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:44.648829 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 18:57:44.650101 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:44.649688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:47463)
0619 18:57:44.649857 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:44.649941 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:57:45.015367 32545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:57:45.015703 32545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:57:45.015884 32545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:57:45.023674 32545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:57:45.023818 32545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:57:45.024016 32545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:57:45.034505 32545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:38467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=43769
--tserver_master_addrs=127.15.91.62:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:57:45.036021 32545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:57:45.037513 32545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:57:45.053081 32551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:45.053260 32553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:45.053267 32550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:57:45.054399 32545 server_base.cc:1034] running on GCE node
I20260619 18:57:45.055274 32545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:57:45.056461 32545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:57:45.057750 32545 hybrid_clock.cc:648] HybridClock initialized: now 1781895465057692 us; error 61 us; skew 500 ppm
I20260619 18:57:45.058164 32545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:57:45 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895465, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:57:45.061825 32545 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:57:45.063653 32545 webserver.cc:460] Webserver started at http://127.15.91.2:43769/ using document root <none> and password file <none>
I20260619 18:57:45.064419 32545 fs_manager.cc:359] Metadata directory not provided
I20260619 18:57:45.064553 32545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:57:45.071043 32545 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:57:45.074271 32560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:45.076023 32545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:57:45.076261 32545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "c47795cd4d3441a0b2f064543b30c4e1"
format_stamp: "Formatted at 2026-06-19 18:56:58 on dist-test-slave-dm3f"
server_key: "741b30708213ce6a2cfe55fce83bfb09"
server_key_iv: "15034a67d35a1b549fb72c593489c025"
server_key_version: "encryptionkey@0"
I20260619 18:57:45.077541 32545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:57:45.095057 32545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:57:45.100795 32545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:57:45.101076 32545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:57:45.102275 32545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:57:45.103968 32545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:57:45.104086 32545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:45.104257 32545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:57:45.104352 32545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:45.145069 32545 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:38467
I20260619 18:57:45.145116 32672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:38467 every 8 connection(s)
I20260619 18:57:45.147163 32545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:57:45.150041 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 32545
I20260619 18:57:45.150976 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:39201
--local_ip_for_outbound_sockets=127.15.91.3
--tserver_master_addrs=127.15.91.62:46783
--webserver_port=43849
--webserver_interface=127.15.91.3
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:57:45.154395 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:45.151565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:46643)
0619 18:57:45.152620 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:45.152935 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":446,"threads_started":1}
W20260619 18:57:45.155404 32673 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:46783 (0 consecutive failures): Network error: Failed to ping master at 127.15.91.62:46783: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
I20260619 18:57:45.156646 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:45.156083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:45493)
0619 18:57:45.156326 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:45.156453 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 18:57:45.158018 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:45.157480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:58257)
0619 18:57:45.157694 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:45.157818 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 18:57:45.158488 32673 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:57:45.524930 32676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:57:45.525300 32676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:57:45.525513 32676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:57:45.533994 32676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:57:45.534154 32676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:57:45.534355 32676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:57:45.545452 32676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:39201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=43849
--tserver_master_addrs=127.15.91.62:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:57:45.547122 32676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:57:45.548682 32676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:57:45.563920 32684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:45.564050 32682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:57:45.564026 32681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:57:45.564996 32676 server_base.cc:1034] running on GCE node
I20260619 18:57:45.565850 32676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:57:45.567159 32676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:57:45.568439 32676 hybrid_clock.cc:648] HybridClock initialized: now 1781895465568389 us; error 56 us; skew 500 ppm
I20260619 18:57:45.568830 32676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:57:45 dist-test-slave-dm3f krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895465, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:57:45.572403 32676 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:57:45.574247 32676 webserver.cc:460] Webserver started at http://127.15.91.3:43849/ using document root <none> and password file <none>
I20260619 18:57:45.575146 32676 fs_manager.cc:359] Metadata directory not provided
I20260619 18:57:45.575278 32676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:57:45.581833 32676 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 18:57:45.585253 32691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:45.587261 32676 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:57:45.587518 32676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "3868e0574b874913bb7d7a0a0953fb39"
format_stamp: "Formatted at 2026-06-19 18:56:59 on dist-test-slave-dm3f"
server_key: "cd60e9c3febb477df37c7aff3417c503"
server_key_iv: "6548cc574ecc14e3dce9b5b0640b3926"
server_key_version: "encryptionkey@0"
I20260619 18:57:45.588716 32676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:57:45.615113 32676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:57:45.620386 32676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:57:45.620724 32676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:57:45.621953 32676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:57:45.623678 32676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:57:45.623799 32676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:45.623952 32676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:57:45.624058 32676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:57:45.652249 300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:45.651245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:44613)
0619 18:57:45.651809 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:45.651916 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":265,"threads_started":1}
I20260619 18:57:45.664772 32676 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:39201
I20260619 18:57:45.664868 336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:39201 every 8 connection(s)
I20260619 18:57:45.666500 32676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:57:45.670661 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 32676
I20260619 18:57:45.672963 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:45.670012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:51785)
0619 18:57:45.671133 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:45.671450 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":397,"threads_started":1}
W20260619 18:57:45.674098 337 heartbeater.cc:644] Failed to heartbeat to 127.15.91.62:46783 (0 consecutive failures): Network error: Failed to ping master at 127.15.91.62:46783: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
I20260619 18:57:45.675318 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:45.674898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:50407)
0619 18:57:45.675080 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:45.675180 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 18:57:45.676506 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:45.676014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:34255)
0619 18:57:45.676251 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:45.676349 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260619 18:57:45.676971 337 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:57:45.806196 341 raft_consensus.cc:482] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:57:45.806535 341 raft_consensus.cc:504] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:45.808488 341 leader_election.cc:288] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201), c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
Jun 19 18:57:45 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895464, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
Jun 19 18:57:45 dist-test-slave-dm3f krb5kdc[31218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895464, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:57:45.840776 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:45.809023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:45383 (local address 127.15.91.3:39201)
0619 18:57:45.816015 (+ 6992us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:45.816026 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:45.816072 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:57:45.818606 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:45.818621 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:45.819341 (+ 720us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:45.819817 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:45.822181 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:45.823611 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:45.826088 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:45.827083 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:45.827292 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:45.832138 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:45.832147 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:45.832164 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:45.832219 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:45.835837 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:45.836949 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:45.836963 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:45.836972 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:45.837257 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:45.837987 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:45.837991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:45.837993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:45.838635 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:45.838919 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:45.839824 (+ 905us) server_negotiation.cc:299] Negotiation successful
0619 18:57:45.840274 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6793,"thread_start_us":6607,"threads_started":1}
I20260619 18:57:45.840785 344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:45.809788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:47153 (local address 127.15.91.2:38467)
0619 18:57:45.815580 (+ 5792us) server_negotiation.cc:206] Beginning negotiation
0619 18:57:45.815609 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 18:57:45.816305 (+ 696us) server_negotiation.cc:407] Connection header received
0619 18:57:45.818606 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:57:45.818621 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:57:45.819341 (+ 720us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:57:45.819817 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:57:45.822161 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:45.823606 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:45.826072 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:57:45.827072 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:57:45.827327 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:45.832203 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:57:45.832212 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:57:45.832226 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:57:45.832271 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:57:45.836426 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:45.837529 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:45.837539 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:45.837547 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:45.837858 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:57:45.838321 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:57:45.838327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:57:45.838330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:57:45.838976 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:57:45.839189 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 18:57:45.839827 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 18:57:45.840323 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5497,"thread_start_us":5296,"threads_started":1}
I20260619 18:57:45.840816 343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:45.809281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38467 (local address 127.15.91.1:47153)
0619 18:57:45.816140 (+ 6859us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:45.816159 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:57:45.816314 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:57:45.820521 (+ 4207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:57:45.820524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:57:45.821063 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:57:45.821845 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:45.821875 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:45.823898 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:45.823903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:45.825750 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:45.825773 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:45.827612 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:45.827617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:45.827772 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:45.828698 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:57:45.828717 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:57:45.831858 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:57:45.836745 (+ 4887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:57:45.836751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:57:45.836755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:57:45.837280 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:57:45.838023 (+ 743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:57:45.838027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:57:45.838030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:57:45.838191 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:57:45.839128 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:57:45.839133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:57:45.839303 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 18:57:45.839558 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 18:57:45.840296 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6723,"mutex_wait_us":322,"spinlock_wait_cycles":68352,"thread_start_us":6550,"threads_started":1}
I20260619 18:57:45.841365 300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:45.808821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:39201 (local address 127.15.91.1:45383)
0619 18:57:45.809029 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:45.809073 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:57:45.809712 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:57:45.820069 (+ 10357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:57:45.820079 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:57:45.820875 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:57:45.821846 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:45.821875 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:45.823912 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:45.823916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:45.825751 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:57:45.825774 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:57:45.827356 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:57:45.827365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:57:45.827692 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:57:45.828668 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:57:45.828696 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:57:45.831714 (+ 3018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:57:45.836109 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:57:45.836118 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:57:45.836134 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:57:45.836740 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:57:45.837555 (+ 815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:57:45.837559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:57:45.837561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:57:45.837836 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:57:45.838856 (+ 1020us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:57:45.838865 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:57:45.839237 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 18:57:45.839558 (+ 321us) client_negotiation.cc:240] Negotiation successful
0619 18:57:45.840492 (+ 934us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":449}
W20260619 18:57:45.845188 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
W20260619 18:57:45.845664 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
I20260619 18:57:45.845870 32427 leader_election.cc:302] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e68a91804f3473aa67cfacd693e4c58; no voters: 3868e0574b874913bb7d7a0a0953fb39, c47795cd4d3441a0b2f064543b30c4e1
I20260619 18:57:45.846371 341 raft_consensus.cc:2726] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:57:46.160274 345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:46.159219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:47601)
0619 18:57:46.159836 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:46.159994 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":301,"threads_started":1}
I20260619 18:57:46.654263 346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:46.653410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:50477)
0619 18:57:46.653970 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:46.654073 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":281,"threads_started":1}
I20260619 18:57:46.678530 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:46.677673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:38005)
0619 18:57:46.678165 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:46.678290 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
I20260619 18:57:47.162209 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:47.161464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:34045)
0619 18:57:47.161940 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:47.162040 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":242,"threads_started":1}
I20260619 18:57:47.656250 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:47.655302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:59629)
0619 18:57:47.655926 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:47.656032 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":287,"threads_started":1}
I20260619 18:57:47.680662 350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:47.679847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:54979)
0619 18:57:47.680359 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:47.680465 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":278,"threads_started":1}
I20260619 18:57:48.046325 351 raft_consensus.cc:482] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:57:48.046572 351 raft_consensus.cc:504] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:48.047420 351 leader_election.cc:288] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201), c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
W20260619 18:57:48.048596 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
W20260619 18:57:48.048848 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
I20260619 18:57:48.048936 32427 leader_election.cc:302] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e68a91804f3473aa67cfacd693e4c58; no voters: 3868e0574b874913bb7d7a0a0953fb39, c47795cd4d3441a0b2f064543b30c4e1
I20260619 18:57:48.049247 351 raft_consensus.cc:2726] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:57:48.164201 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:48.163438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:51587)
0619 18:57:48.163934 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:48.164028 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":257,"threads_started":1}
I20260619 18:57:48.658248 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:48.657423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:55245)
0619 18:57:48.657945 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:48.658045 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":273,"threads_started":1}
I20260619 18:57:48.682686 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:48.681701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:35537)
0619 18:57:48.682316 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:48.682465 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":341,"threads_started":1}
I20260619 18:57:49.166260 355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:49.165333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:40569)
0619 18:57:49.165951 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:49.166050 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":365,"threads_started":1}
I20260619 18:57:49.660526 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:49.659580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:33691)
0619 18:57:49.660218 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:49.660318 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":290,"threads_started":1}
I20260619 18:57:49.684746 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:49.683783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:46389)
0619 18:57:49.684315 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:49.684493 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":288,"threads_started":1}
I20260619 18:57:50.168367 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:50.167482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:48885)
0619 18:57:50.168084 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:50.168188 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":315,"threads_started":1}
I20260619 18:57:50.283195 359 raft_consensus.cc:482] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:57:50.283427 359 raft_consensus.cc:504] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:50.284169 359 leader_election.cc:288] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201), c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
W20260619 18:57:50.285130 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
W20260619 18:57:50.285341 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
I20260619 18:57:50.285423 32427 leader_election.cc:302] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e68a91804f3473aa67cfacd693e4c58; no voters: 3868e0574b874913bb7d7a0a0953fb39, c47795cd4d3441a0b2f064543b30c4e1
I20260619 18:57:50.285712 359 raft_consensus.cc:2726] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:57:50.662621 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:50.661578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:37201)
0619 18:57:50.662226 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:50.662366 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":366,"threads_started":1}
I20260619 18:57:50.686455 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:50.685736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:51509)
0619 18:57:50.686184 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:50.686282 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":244,"threads_started":1}
I20260619 18:57:51.170415 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:51.169549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:39525)
0619 18:57:51.170115 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:51.170218 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":308,"threads_started":1}
I20260619 18:57:51.664490 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:51.663690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:59079)
0619 18:57:51.664175 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:51.664269 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
I20260619 18:57:51.688529 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:51.687633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:33139)
0619 18:57:51.688195 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:51.688301 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":299,"threads_started":1}
I20260619 18:57:52.172485 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:52.171591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:41779)
0619 18:57:52.172162 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:52.172261 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":282,"threads_started":1}
I20260619 18:57:52.295058 366 raft_consensus.cc:482] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:57:52.295296 366 raft_consensus.cc:504] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:52.296057 366 leader_election.cc:288] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201), c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
W20260619 18:57:52.297168 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
W20260619 18:57:52.297444 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
I20260619 18:57:52.297531 32427 leader_election.cc:302] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e68a91804f3473aa67cfacd693e4c58; no voters: 3868e0574b874913bb7d7a0a0953fb39, c47795cd4d3441a0b2f064543b30c4e1
I20260619 18:57:52.297864 366 raft_consensus.cc:2726] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:57:52.666584 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:52.665676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:57619)
0619 18:57:52.666237 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:52.666363 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":293,"threads_started":1}
I20260619 18:57:52.690897 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:52.689761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:57875)
0619 18:57:52.690402 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:52.690498 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":322,"threads_started":1}
I20260619 18:57:53.174523 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:53.173693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:33081)
0619 18:57:53.174215 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:53.174311 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":277,"threads_started":1}
I20260619 18:57:53.668802 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:53.667839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:39519)
0619 18:57:53.668482 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:53.668595 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":370,"threads_started":1}
I20260619 18:57:53.693159 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:53.692244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:52697)
0619 18:57:53.692876 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:53.692970 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":383,"threads_started":1}
I20260619 18:57:54.124823 372 raft_consensus.cc:482] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:57:54.125121 372 raft_consensus.cc:504] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:57:54.125821 372 leader_election.cc:288] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201), c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
W20260619 18:57:54.127173 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
W20260619 18:57:54.127442 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
I20260619 18:57:54.127583 32427 leader_election.cc:302] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e68a91804f3473aa67cfacd693e4c58; no voters: 3868e0574b874913bb7d7a0a0953fb39, c47795cd4d3441a0b2f064543b30c4e1
I20260619 18:57:54.127880 372 raft_consensus.cc:2726] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:57:54.176571 373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:54.175773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:54939)
0619 18:57:54.176298 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:54.176401 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":260,"threads_started":1}
I20260619 18:57:54.671408 374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:54.670369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:38033)
0619 18:57:54.671122 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:54.671213 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":371,"threads_started":1}
I20260619 18:57:54.695217 375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:54.694309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:56361)
0619 18:57:54.694900 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:54.695039 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":274,"threads_started":1}
I20260619 18:57:55.178580 376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:55.177768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:46035)
0619 18:57:55.178283 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:55.178380 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":266,"threads_started":1}
I20260619 18:57:55.673436 377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:55.672625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:59411)
0619 18:57:55.673154 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:55.673246 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":286,"threads_started":1}
I20260619 18:57:55.697381 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:55.696358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:58147)
0619 18:57:55.696906 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:55.697031 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":307,"threads_started":1}
I20260619 18:57:56.180630 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:56.179711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:52121)
0619 18:57:56.180312 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:56.180422 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":297,"threads_started":1}
I20260619 18:57:56.675755 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:56.674680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:38195)
0619 18:57:56.675346 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:56.675450 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":285,"threads_started":1}
I20260619 18:57:56.699723 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:56.698805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:34507)
0619 18:57:56.699451 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:56.699546 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":368,"threads_started":1}
I20260619 18:57:57.182991 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:57.181879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:41149)
0619 18:57:57.182588 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:57.182763 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":341,"threads_started":1}
I20260619 18:57:57.678037 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:57.677062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:55383)
0619 18:57:57.677729 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:57.677836 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":343,"threads_started":1}
I20260619 18:57:57.702090 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:57.701070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:37983)
0619 18:57:57.701741 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:57.701861 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":344,"threads_started":1}
I20260619 18:57:58.185177 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:58.184313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:43329)
0619 18:57:58.184868 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:58.184980 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":297,"threads_started":1}
I20260619 18:57:58.680269 386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:58.679288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:43137)
0619 18:57:58.679840 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:58.679937 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":298,"threads_started":1}
I20260619 18:57:58.704264 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:58.703368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:54619)
0619 18:57:58.703950 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:58.704089 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":329,"threads_started":1}
I20260619 18:57:59.187698 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:59.186401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:41193)
0619 18:57:59.187202 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:59.187361 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":579,"threads_started":1}
I20260619 18:57:59.683593 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:59.681851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:35669)
0619 18:57:59.683161 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:59.683330 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1155,"thread_start_us":987,"threads_started":1}
I20260619 18:57:59.706470 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:57:59.705585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:47677)
0619 18:57:59.706171 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 18:57:59.706266 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":348,"threads_started":1}
I20260619 18:58:00.190092 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:00.189254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.2:36953)
0619 18:58:00.189739 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:00.189890 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":245,"threads_started":1}
I20260619 18:58:00.193352 392 raft_consensus.cc:482] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:58:00.193608 392 raft_consensus.cc:504] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e68a91804f3473aa67cfacd693e4c58" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 42901 } } peers { permanent_uuid: "3868e0574b874913bb7d7a0a0953fb39" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39201 } } peers { permanent_uuid: "c47795cd4d3441a0b2f064543b30c4e1" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38467 } }
I20260619 18:58:00.194362 392 leader_election.cc:288] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201), c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467)
W20260619 18:58:00.195523 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c47795cd4d3441a0b2f064543b30c4e1 (127.15.91.2:38467): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
W20260619 18:58:00.195802 32427 leader_election.cc:341] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3868e0574b874913bb7d7a0a0953fb39 (127.15.91.3:39201): Not found: Tablet not found: d585c27d51cd4b018bb1111de519d3ed
I20260619 18:58:00.195926 32427 leader_election.cc:302] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e68a91804f3473aa67cfacd693e4c58; no voters: 3868e0574b874913bb7d7a0a0953fb39, c47795cd4d3441a0b2f064543b30c4e1
I20260619 18:58:00.196234 392 raft_consensus.cc:2726] T d585c27d51cd4b018bb1111de519d3ed P 8e68a91804f3473aa67cfacd693e4c58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:58:00.672616 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 32370
I20260619 18:58:00.685732 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:00.684673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.1:36061)
0619 18:58:00.685339 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:00.685467 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":331,"threads_started":1}
I20260619 18:58:00.698501 393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:00.674200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.15.91.1:42901)
0619 18:58:00.674937 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:00.674950 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:00.674998 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:58:00.675259 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:00.675273 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:00.675644 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:00.675834 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:00.677479 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:00.678694 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:00.680552 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:00.681199 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:00.681378 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:00.690814 (+ 9436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:00.690838 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:00.690857 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:00.690906 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:00.694604 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:00.695682 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:00.695691 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:00.695694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:00.695877 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:00.696385 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:00.696399 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:00.696402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:00.696874 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:00.697342 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:00.697720 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 18:58:00.698068 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":312,"threads_started":1}
I20260619 18:58:00.700022 32536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:00.708583 396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:00.707705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46783 (local address 127.15.91.3:55825)
0619 18:58:00.708281 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:00.708414 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.62:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":307,"threads_started":1}
I20260619 18:58:00.796512 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 32370
I20260619 18:58:00.815757 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 32545
I20260619 18:58:00.831796 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:00.816756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.15.91.2:38467)
0619 18:58:00.817322 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:00.817327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:00.817346 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:58:00.817405 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:00.817409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:00.817473 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:00.817577 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:00.819164 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:00.820047 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:00.821517 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:00.822266 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:00.822600 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:00.825697 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:00.825704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:00.825708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:00.825766 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:00.828844 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:00.829717 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:00.829722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:00.829725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:00.829885 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:00.830288 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:00.830292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:00.830294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:00.830595 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:00.830899 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:00.831268 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 18:58:00.831486 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":350,"threads_started":1}
I20260619 18:58:00.832733 32668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:00.931633 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 32545
I20260619 18:58:00.950415 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 32676
I20260619 18:58:00.970427 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:00.951474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.15.91.3:39201)
0619 18:58:00.952173 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:00.952178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:00.952199 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:58:00.952256 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:00.952260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:00.952332 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:00.952439 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:00.954027 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:00.955092 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:00.957142 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:00.957793 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:00.957988 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:00.961902 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:00.961909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:00.961913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:00.961995 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:00.965537 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:00.966981 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:00.966989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:00.966992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:00.967484 (+ 492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:00.968297 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:00.968302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:00.968305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:00.968706 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:00.968979 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:00.969537 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 18:58:00.969787 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":402,"threads_started":1}
I20260619 18:58:00.971378 332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:01.063531 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 32676
2026-06-19T18:58:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65376 ms)
[----------] 30 tests from SecurityITest (316472 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:01 dist-test-slave-dm3f krb5kdc[403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:01 dist-test-slave-dm3f krb5kdc[403](info): set up 2 sockets
Jun 19 18:58:01 dist-test-slave-dm3f krb5kdc[403](info): commencing operation
krb5kdc: starting...
W20260619 18:58:02.207693 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:02 dist-test-slave-dm3f krb5kdc[403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:58:02.478477 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:39011
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:39011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:02.847739 416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:02.848181 416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:02.848354 416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:02.857568 416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:02.857762 416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:02.857847 416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:02.857967 416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:02.858057 416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:02.869174 416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:39011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:39011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:02.870683 416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:02.872270 416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:02.885878 424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:02.885983 422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:02.885939 421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:02.887621 416 server_base.cc:1034] running on GCE node
I20260619 18:58:02.888954 416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:02.890854 416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:02.892140 416 hybrid_clock.cc:648] HybridClock initialized: now 1781895482892087 us; error 44 us; skew 500 ppm
I20260619 18:58:02.892609 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:02 dist-test-slave-dm3f krb5kdc[403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895482, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:02.896526 416 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:58:02.898608 416 webserver.cc:460] Webserver started at http://127.15.91.62:35971/ using document root <none> and password file <none>
I20260619 18:58:02.899534 416 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:02.899710 416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:02.900081 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:02.903635 416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "06e085dad19c430cbdfdedbae47451e4"
format_stamp: "Formatted at 2026-06-19 18:58:02 on dist-test-slave-dm3f"
server_key: "f4fba89652e9f41b3cf85dae9f733493"
server_key_iv: "410175859ff6a1bb156021bf864a792e"
server_key_version: "encryptionkey@0"
I20260619 18:58:02.904706 416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "06e085dad19c430cbdfdedbae47451e4"
format_stamp: "Formatted at 2026-06-19 18:58:02 on dist-test-slave-dm3f"
server_key: "f4fba89652e9f41b3cf85dae9f733493"
server_key_iv: "410175859ff6a1bb156021bf864a792e"
server_key_version: "encryptionkey@0"
I20260619 18:58:02.912034 416 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260619 18:58:02.916795 431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:02.918622 416 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:58:02.918996 416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "06e085dad19c430cbdfdedbae47451e4"
format_stamp: "Formatted at 2026-06-19 18:58:02 on dist-test-slave-dm3f"
server_key: "f4fba89652e9f41b3cf85dae9f733493"
server_key_iv: "410175859ff6a1bb156021bf864a792e"
server_key_version: "encryptionkey@0"
I20260619 18:58:02.919281 416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:02.933640 416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:02.938935 416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:02.939289 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:02.961328 416 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:39011
I20260619 18:58:02.961553 482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:39011 every 8 connection(s)
I20260619 18:58:02.963152 416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:02.966194 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 416
I20260619 18:58:02.966450 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:02.967252 15724 external_mini_cluster.cc:1428] Setting key ded182bc78c3de3116d27784b5591eb9
I20260619 18:58:02.968077 483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:02.983637 483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4: Bootstrap starting.
Jun 19 18:58:02 dist-test-slave-dm3f krb5kdc[403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:02.987653 483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:02.989100 483 log.cc:824] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:02.992600 483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4: No bootstrap required, opened a new log
I20260619 18:58:02.995148 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:02.969626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.15.91.62:39011)
0619 18:58:02.970622 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:02.970644 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:02.970707 (+ 63us) server_negotiation.cc:407] Connection header received
0619 18:58:02.974196 (+ 3489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:02.974232 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:02.974904 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:02.975423 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:02.977154 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:02.978578 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:02.980164 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:02.981040 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:02.981314 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:02.986265 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:02.986274 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:02.986291 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:02.986339 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:02.990102 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:02.991141 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:02.991150 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:02.991158 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:02.991371 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:02.991832 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:02.991838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:02.991842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:02.992449 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:02.992810 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:02.993385 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 18:58:02.993885 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":443,"threads_started":1}
I20260619 18:58:03.001592 483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e085dad19c430cbdfdedbae47451e4" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39011 } }
I20260619 18:58:03.002115 483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:03.002236 483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e085dad19c430cbdfdedbae47451e4, State: Initialized, Role: FOLLOWER
I20260619 18:58:03.003074 483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e085dad19c430cbdfdedbae47451e4" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39011 } }
I20260619 18:58:03.003357 483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:03.003526 483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:03.003731 483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:03.006850 483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e085dad19c430cbdfdedbae47451e4" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39011 } }
I20260619 18:58:03.007537 483 leader_election.cc:302] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06e085dad19c430cbdfdedbae47451e4; no voters:
I20260619 18:58:03.008293 483 leader_election.cc:288] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:03.008461 488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:03.009886 488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [term 1 LEADER]: Becoming Leader. State: Replica: 06e085dad19c430cbdfdedbae47451e4, State: Running, Role: LEADER
I20260619 18:58:03.010527 488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e085dad19c430cbdfdedbae47451e4" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39011 } }
I20260619 18:58:03.010926 483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:03.013746 490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06e085dad19c430cbdfdedbae47451e4. Latest consensus state: current_term: 1 leader_uuid: "06e085dad19c430cbdfdedbae47451e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e085dad19c430cbdfdedbae47451e4" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39011 } } }
I20260619 18:58:03.014312 490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:03.013813 489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06e085dad19c430cbdfdedbae47451e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e085dad19c430cbdfdedbae47451e4" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39011 } } }
I20260619 18:58:03.014956 489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:03.017530 496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:03.023844 496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:03.035005 496 catalog_manager.cc:1347] Generated new cluster ID: f8177ba273b441b08a23608ef9eab66e
I20260619 18:58:03.035183 496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:03.048326 496 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:03.050555 496 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:03.061050 496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4: Generated new TSK 0
I20260619 18:58:03.062093 496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:58:03.139078 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39011
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:03.544386 509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:03.544714 509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:03.544890 509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:03.552945 509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:03.553094 509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:03.553300 509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:58:03.564433 509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:03.565969 509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:03.567678 509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:03.583238 514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:03.583238 515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:03.583495 517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:03.584543 509 server_base.cc:1034] running on GCE node
I20260619 18:58:03.585338 509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:03.586460 509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:03.587723 509 hybrid_clock.cc:648] HybridClock initialized: now 1781895483587636 us; error 84 us; skew 500 ppm
I20260619 18:58:03.588164 509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:03 dist-test-slave-dm3f krb5kdc[403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:03.591908 509 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:58:03.593695 509 webserver.cc:460] Webserver started at http://127.15.91.1:32837/ using document root <none> and password file <none>
I20260619 18:58:03.594460 509 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:03.594578 509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:03.594973 509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:03.598201 509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "232f796689934f76a9ee047e5c0fa28e"
format_stamp: "Formatted at 2026-06-19 18:58:03 on dist-test-slave-dm3f"
server_key: "c8f680cb4b488293e3ae456df4d05d29"
server_key_iv: "ec61447b73aa0d15024cec21d1725e6b"
server_key_version: "encryptionkey@0"
I20260619 18:58:03.599144 509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "232f796689934f76a9ee047e5c0fa28e"
format_stamp: "Formatted at 2026-06-19 18:58:03 on dist-test-slave-dm3f"
server_key: "c8f680cb4b488293e3ae456df4d05d29"
server_key_iv: "ec61447b73aa0d15024cec21d1725e6b"
server_key_version: "encryptionkey@0"
I20260619 18:58:03.605469 509 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:58:03.610013 524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:03.611966 509 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:58:03.612262 509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "232f796689934f76a9ee047e5c0fa28e"
format_stamp: "Formatted at 2026-06-19 18:58:03 on dist-test-slave-dm3f"
server_key: "c8f680cb4b488293e3ae456df4d05d29"
server_key_iv: "ec61447b73aa0d15024cec21d1725e6b"
server_key_version: "encryptionkey@0"
I20260619 18:58:03.612560 509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:03.631759 509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:03.637009 509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:03.637405 509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:03.638562 509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:03.640318 509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:03.640453 509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:03.640627 509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:03.640707 509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:03.679072 509 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:37841
I20260619 18:58:03.679114 636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:37841 every 8 connection(s)
I20260619 18:58:03.680599 509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:58:03.682228 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 509
I20260619 18:58:03.682476 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:58:03.683267 15724 external_mini_cluster.cc:1428] Setting key e2dcaae16162a8b9c9846f47defa7703
Jun 19 18:58:03 dist-test-slave-dm3f krb5kdc[403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:03.709800 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:03.683883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:49111 (local address 127.15.91.62:39011)
0619 18:58:03.684449 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:03.684457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:03.685943 (+ 1486us) server_negotiation.cc:407] Connection header received
0619 18:58:03.689334 (+ 3391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:03.689340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:03.689431 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:03.689593 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:03.692316 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:03.693124 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:03.696272 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:03.696721 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:03.696894 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:03.701671 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:03.701677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:03.701681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:03.701723 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:03.704783 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:03.706193 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:03.706199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:03.706201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:03.706365 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:03.707133 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:03.707137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:03.707139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:03.707469 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:03.707612 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:03.709060 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0619 18:58:03.709417 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":276,"threads_started":1}
I20260619 18:58:03.711333 640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:03.684586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39011 (local address 127.15.91.1:49111)
0619 18:58:03.685698 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:03.685803 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:03.688817 (+ 3014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:03.689925 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:03.689941 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:03.690967 (+ 1026us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:03.691965 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:03.692000 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:03.693427 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:03.693437 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:03.696026 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:03.696050 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:03.697061 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:03.697070 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:03.697346 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:03.698177 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:58:03.698209 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:03.701290 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:03.705044 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:03.705060 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:03.705087 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:03.705920 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:03.706526 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:03.706535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:03.706542 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:03.706922 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:03.707642 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:03.707655 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:03.708286 (+ 631us) client_negotiation.cc:769] Sending connection context
0619 18:58:03.708690 (+ 404us) client_negotiation.cc:240] Negotiation successful
0619 18:58:03.709457 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":406,"threads_started":1}
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:58:03.714145 637 heartbeater.cc:347] Connected to a master server at 127.15.91.62:39011
I20260619 18:58:03.714581 637 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:03.715554 637 heartbeater.cc:511] Master 127.15.91.62:39011 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:58:03.718669 448 ts_manager.cc:194] Registered new tserver with Master: 232f796689934f76a9ee047e5c0fa28e (127.15.91.1:37841)
I20260619 18:58:03.720543 448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:49111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:58:03.746891 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39011
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:04.120656 643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:04.120994 643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:04.121097 643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:04.129213 643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:04.129365 643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:04.129485 643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:58:04.139991 643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:04.141381 643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:04.142863 643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:04.158063 649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:04.157966 651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:04.158082 648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:04.164678 643 server_base.cc:1034] running on GCE node
I20260619 18:58:04.165637 643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:04.167069 643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:04.168447 643 hybrid_clock.cc:648] HybridClock initialized: now 1781895484168369 us; error 67 us; skew 500 ppm
I20260619 18:58:04.168924 643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:04 dist-test-slave-dm3f krb5kdc[403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895484, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:04.172919 643 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:58:04.175048 643 webserver.cc:460] Webserver started at http://127.15.91.2:38783/ using document root <none> and password file <none>
I20260619 18:58:04.175961 643 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:04.176151 643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:04.176514 643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:04.180117 643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "f053fb6252874522a9d7946b743f7d38"
format_stamp: "Formatted at 2026-06-19 18:58:04 on dist-test-slave-dm3f"
server_key: "bb93293ef0930463e3a22629049dc4c7"
server_key_iv: "cfd78226ba2c51391363f76a626f571f"
server_key_version: "encryptionkey@0"
I20260619 18:58:04.181167 643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "f053fb6252874522a9d7946b743f7d38"
format_stamp: "Formatted at 2026-06-19 18:58:04 on dist-test-slave-dm3f"
server_key: "bb93293ef0930463e3a22629049dc4c7"
server_key_iv: "cfd78226ba2c51391363f76a626f571f"
server_key_version: "encryptionkey@0"
I20260619 18:58:04.188494 643 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260619 18:58:04.193562 658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:04.196393 643 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260619 18:58:04.196771 643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "f053fb6252874522a9d7946b743f7d38"
format_stamp: "Formatted at 2026-06-19 18:58:04 on dist-test-slave-dm3f"
server_key: "bb93293ef0930463e3a22629049dc4c7"
server_key_iv: "cfd78226ba2c51391363f76a626f571f"
server_key_version: "encryptionkey@0"
I20260619 18:58:04.197116 643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:04.223593 643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:04.229206 643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:04.229576 643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:04.230865 643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:04.232537 643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:04.232685 643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:04.232873 643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:04.232956 643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:04.274916 643 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:44307
I20260619 18:58:04.274933 770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:44307 every 8 connection(s)
I20260619 18:58:04.276503 643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:58:04.286823 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 643
I20260619 18:58:04.287012 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:58:04.287689 15724 external_mini_cluster.cc:1428] Setting key 91b90314dab92e49c9880c032eb7eeed
Jun 19 18:58:04 dist-test-slave-dm3f krb5kdc[403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895484, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:04.304637 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:04.279678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:32949 (local address 127.15.91.62:39011)
0619 18:58:04.280295 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:04.280301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:04.281538 (+ 1237us) server_negotiation.cc:407] Connection header received
0619 18:58:04.283997 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:04.284003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:04.284114 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:04.284244 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:04.286112 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.286985 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.289568 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.290678 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.290861 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:04.296470 (+ 5609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:04.296476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:04.296480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:04.296524 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:04.299717 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:04.301061 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:04.301071 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:04.301078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:04.301291 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:04.301899 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:04.301908 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:04.301914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:04.302403 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:04.302568 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:04.303759 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0619 18:58:04.304106 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":331,"threads_started":1}
I20260619 18:58:04.305408 774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:04.280376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39011 (local address 127.15.91.2:32949)
0619 18:58:04.281304 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:04.281363 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:04.283630 (+ 2267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:04.284536 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:04.284546 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:04.285198 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:04.285860 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:04.285880 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.287274 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.287283 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:04.289358 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:04.289377 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.291053 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.291062 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:04.291441 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:04.292330 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:58:04.292381 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:04.296059 (+ 3678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:04.299969 (+ 3910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:04.299986 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:04.300010 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:04.300852 (+ 842us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:04.301473 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:04.301476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:04.301479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:04.301758 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:04.302690 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:04.302698 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:04.303238 (+ 540us) client_negotiation.cc:769] Sending connection context
0619 18:58:04.303597 (+ 359us) client_negotiation.cc:240] Negotiation successful
0619 18:58:04.304093 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":362,"threads_started":1}
I20260619 18:58:04.307338 771 heartbeater.cc:347] Connected to a master server at 127.15.91.62:39011
I20260619 18:58:04.307724 771 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:04.308578 771 heartbeater.cc:511] Master 127.15.91.62:39011 requested a full tablet report, sending...
I20260619 18:58:04.310396 448 ts_manager.cc:194] Registered new tserver with Master: f053fb6252874522a9d7946b743f7d38 (127.15.91.2:44307)
I20260619 18:58:04.311388 448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:32949
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:58:04.350263 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39011
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:04.716362 777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:04.716688 777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:04.716861 777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 18:58:04.724742 637 heartbeater.cc:503] Master 127.15.91.62:39011 was elected leader, sending a full tablet report...
W20260619 18:58:04.725698 777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:04.725843 777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:04.726073 777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:58:04.737849 777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:04.739470 777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:04.740926 777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:04.757112 782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:04.757233 785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:04.757282 783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:04.758358 777 server_base.cc:1034] running on GCE node
I20260619 18:58:04.759241 777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:04.760558 777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:04.762013 777 hybrid_clock.cc:648] HybridClock initialized: now 1781895484761939 us; error 73 us; skew 500 ppm
I20260619 18:58:04.762485 777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:04 dist-test-slave-dm3f krb5kdc[403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895484, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:04.766937 777 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:58:04.769047 777 webserver.cc:460] Webserver started at http://127.15.91.3:36313/ using document root <none> and password file <none>
I20260619 18:58:04.769870 777 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:04.770008 777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:04.770318 777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:04.773773 777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "6bd3dc715c20460c8bd9300d829c4a7f"
format_stamp: "Formatted at 2026-06-19 18:58:04 on dist-test-slave-dm3f"
server_key: "71db381d14ecf9de027e82faa9590294"
server_key_iv: "2c9280f18b25933a420276b05ddb1381"
server_key_version: "encryptionkey@0"
I20260619 18:58:04.774704 777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "6bd3dc715c20460c8bd9300d829c4a7f"
format_stamp: "Formatted at 2026-06-19 18:58:04 on dist-test-slave-dm3f"
server_key: "71db381d14ecf9de027e82faa9590294"
server_key_iv: "2c9280f18b25933a420276b05ddb1381"
server_key_version: "encryptionkey@0"
I20260619 18:58:04.782094 777 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 18:58:04.787022 792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:04.788915 777 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:04.789149 777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "6bd3dc715c20460c8bd9300d829c4a7f"
format_stamp: "Formatted at 2026-06-19 18:58:04 on dist-test-slave-dm3f"
server_key: "71db381d14ecf9de027e82faa9590294"
server_key_iv: "2c9280f18b25933a420276b05ddb1381"
server_key_version: "encryptionkey@0"
I20260619 18:58:04.789371 777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:04.820345 777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:04.825790 777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:04.826166 777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:04.827438 777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:04.829135 777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:04.829265 777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:04.829429 777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:04.829514 777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:04.868000 777 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:39627
I20260619 18:58:04.868108 905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:39627 every 8 connection(s)
I20260619 18:58:04.869524 777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:58:04.878803 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 777
I20260619 18:58:04.879029 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:58:04.879520 15724 external_mini_cluster.cc:1428] Setting key 5bf112373ec6d3f42854a8d0837328be
Jun 19 18:58:04 dist-test-slave-dm3f krb5kdc[403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895484, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:04.895396 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:04.872804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:46409 (local address 127.15.91.62:39011)
0619 18:58:04.873300 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:04.873305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:04.874614 (+ 1309us) server_negotiation.cc:407] Connection header received
0619 18:58:04.876749 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:04.876754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:04.876838 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:04.876956 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:04.879169 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.880084 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.882568 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.883151 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.883302 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:04.887886 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:04.887892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:04.887896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:04.887937 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:04.890997 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:04.891978 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:04.891983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:04.891985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:04.892132 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:04.892883 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:04.892887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:04.892889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:04.893197 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:04.893386 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:04.894595 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0619 18:58:04.894957 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":259,"threads_started":1}
I20260619 18:58:04.896530 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:04.873416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39011 (local address 127.15.91.3:46409)
0619 18:58:04.874371 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:04.874419 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:04.876362 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:04.877245 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:04.877255 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:04.877918 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:04.878855 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:04.878894 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.880307 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.880340 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:04.882365 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:04.882391 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.883447 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.883458 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:04.883751 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:04.884503 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:58:04.884530 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:04.887515 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:04.891217 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:04.891227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:04.891246 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:04.891812 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:04.892444 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:04.892456 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:04.892460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:04.892730 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:04.893444 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:04.893452 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:04.893922 (+ 470us) client_negotiation.cc:769] Sending connection context
0619 18:58:04.894318 (+ 396us) client_negotiation.cc:240] Negotiation successful
0619 18:58:04.895033 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":343,"threads_started":1}
I20260619 18:58:04.898361 906 heartbeater.cc:347] Connected to a master server at 127.15.91.62:39011
I20260619 18:58:04.898804 906 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:04.899659 906 heartbeater.cc:511] Master 127.15.91.62:39011 requested a full tablet report, sending...
I20260619 18:58:04.901518 448 ts_manager.cc:194] Registered new tserver with Master: 6bd3dc715c20460c8bd9300d829c4a7f (127.15.91.3:39627)
I20260619 18:58:04.902526 448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:46409
I20260619 18:58:04.909428 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:58:04.931119 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:04.915005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.15.91.62:39011)
0619 18:58:04.915395 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:04.915403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:04.915818 (+ 415us) server_negotiation.cc:407] Connection header received
0619 18:58:04.916061 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:04.916065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:04.916150 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:04.916275 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:04.918155 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.918994 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.920492 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.920950 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.921102 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:04.924170 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:04.924176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:04.924180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:04.924224 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:04.927463 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:04.928382 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:04.928387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:04.928389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:04.928551 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:04.929015 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:04.929018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:04.929020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:04.929406 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:04.929624 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:04.930398 (+ 774us) server_negotiation.cc:299] Negotiation successful
0619 18:58:04.930666 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 18:58:04.937789 448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:58:04.945570 448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:58:04.996372 921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:04.960706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:44307 (local address 127.0.0.1:40846)
0619 18:58:04.976921 (+ 16215us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:04.976989 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:04.977234 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:04.985078 (+ 7844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:04.985087 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:04.985122 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:04.985664 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:04.985685 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.988416 (+ 2731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.988426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:04.992311 (+ 3885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:04.992341 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.995056 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.995064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:04.995355 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:04.995417 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 18:58:04.995707 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 18:58:04.995949 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15854,"mutex_wait_us":140,"threads_started":1}
I20260619 18:58:04.997423 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:04.960664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.15.91.2:44307)
0619 18:58:04.983793 (+ 23129us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:04.983808 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:04.983873 (+ 65us) server_negotiation.cc:407] Connection header received
0619 18:58:04.984065 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:04.984079 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:04.984453 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:04.984734 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:04.985897 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.988135 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.992636 (+ 4501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.994409 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.994587 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:04.994857 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:04.996825 (+ 1968us) server_negotiation.cc:299] Negotiation successful
0619 18:58:04.997083 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22935,"thread_start_us":22809,"threads_started":1}
I20260619 18:58:04.999866 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:04.962293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:39627 (local address 127.0.0.1:60264)
0619 18:58:04.984834 (+ 22541us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:04.984868 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:04.985081 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:04.986327 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:04.986335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:04.986374 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:04.986901 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:04.986917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.989222 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.989227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:04.993772 (+ 4545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:04.993807 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.998830 (+ 5023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.998837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:04.999073 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:04.999113 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:58:04.999283 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 18:58:04.999479 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22314,"thread_start_us":13901,"threads_started":1}
I20260619 18:58:05.000036 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:04.962566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.15.91.3:39627)
0619 18:58:04.971090 (+ 8524us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:04.971105 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:04.985022 (+ 13917us) server_negotiation.cc:407] Connection header received
0619 18:58:04.985290 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:04.985303 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:04.985705 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:04.985953 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:04.987184 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.988989 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.994122 (+ 5133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:04.996598 (+ 2476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:04.996781 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:04.997189 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:04.999425 (+ 2236us) server_negotiation.cc:299] Negotiation successful
0619 18:58:04.999657 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8197,"thread_start_us":7963,"threads_started":1}
I20260619 18:58:05.003636 706 tablet_service.cc:1467] Processing CreateTablet for tablet 1753caf4c8cf4f3ba7d1d4675d7db750 (DEFAULT_TABLE table=test-table [id=d8245e47768346f6a850c102ebb72d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:05.004674 841 tablet_service.cc:1467] Processing CreateTablet for tablet 1753caf4c8cf4f3ba7d1d4675d7db750 (DEFAULT_TABLE table=test-table [id=d8245e47768346f6a850c102ebb72d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:05.005916 706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1753caf4c8cf4f3ba7d1d4675d7db750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:05.006475 841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1753caf4c8cf4f3ba7d1d4675d7db750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:05.007856 921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:04.968319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:37841 (local address 127.0.0.1:44238)
0619 18:58:04.998070 (+ 29751us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:04.998094 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:04.998254 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:04.999722 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:04.999728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:04.999758 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:05.000086 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:05.000113 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.002442 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.002449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:05.004612 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:05.004638 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.006906 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.006913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:05.007126 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:05.007175 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 18:58:05.007319 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 18:58:05.007501 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29620,"thread_start_us":16011,"threads_started":1}
I20260619 18:58:05.007879 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:04.969169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.15.91.1:37841)
0619 18:58:04.986739 (+ 17570us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:04.986752 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:04.998217 (+ 11465us) server_negotiation.cc:407] Connection header received
0619 18:58:04.998811 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:04.998822 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:04.999180 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:04.999421 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:05.000375 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.002159 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.004955 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.006635 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.006861 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:05.007128 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:05.007375 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 18:58:05.007566 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17312,"thread_start_us":17568,"threads_started":1}
I20260619 18:58:05.012172 572 tablet_service.cc:1467] Processing CreateTablet for tablet 1753caf4c8cf4f3ba7d1d4675d7db750 (DEFAULT_TABLE table=test-table [id=d8245e47768346f6a850c102ebb72d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:05.014295 572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1753caf4c8cf4f3ba7d1d4675d7db750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:05.024482 926 tablet_bootstrap.cc:492] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: Bootstrap starting.
I20260619 18:58:05.027828 925 tablet_bootstrap.cc:492] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38: Bootstrap starting.
I20260619 18:58:05.030843 926 tablet_bootstrap.cc:654] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:05.031148 927 tablet_bootstrap.cc:492] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: Bootstrap starting.
I20260619 18:58:05.032682 926 log.cc:824] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:05.033002 925 tablet_bootstrap.cc:654] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:05.035121 925 log.cc:824] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:05.036377 927 tablet_bootstrap.cc:654] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:05.037770 926 tablet_bootstrap.cc:492] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: No bootstrap required, opened a new log
I20260619 18:58:05.038131 926 ts_tablet_manager.cc:1397] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.005s
I20260619 18:58:05.038394 927 log.cc:824] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:05.039573 925 tablet_bootstrap.cc:492] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38: No bootstrap required, opened a new log
I20260619 18:58:05.039932 925 ts_tablet_manager.cc:1397] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260619 18:58:05.042439 927 tablet_bootstrap.cc:492] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: No bootstrap required, opened a new log
I20260619 18:58:05.042893 927 ts_tablet_manager.cc:1397] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: Time spent bootstrapping tablet: real 0.012s user 0.012s sys 0.000s
I20260619 18:58:05.046811 926 raft_consensus.cc:348] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f053fb6252874522a9d7946b743f7d38" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44307 } } peers { permanent_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39627 } } peers { permanent_uuid: "232f796689934f76a9ee047e5c0fa28e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 37841 } }
I20260619 18:58:05.048091 926 raft_consensus.cc:374] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:05.048251 926 raft_consensus.cc:729] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd3dc715c20460c8bd9300d829c4a7f, State: Initialized, Role: FOLLOWER
I20260619 18:58:05.049039 926 consensus_queue.cc:260] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f053fb6252874522a9d7946b743f7d38" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44307 } } peers { permanent_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39627 } } peers { permanent_uuid: "232f796689934f76a9ee047e5c0fa28e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 37841 } }
I20260619 18:58:05.050417 906 heartbeater.cc:503] Master 127.15.91.62:39011 was elected leader, sending a full tablet report...
I20260619 18:58:05.051301 926 ts_tablet_manager.cc:1428] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
I20260619 18:58:05.052618 925 raft_consensus.cc:348] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f053fb6252874522a9d7946b743f7d38" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44307 } } peers { permanent_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39627 } } peers { permanent_uuid: "232f796689934f76a9ee047e5c0fa28e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 37841 } }
I20260619 18:58:05.053247 925 raft_consensus.cc:374] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:05.053421 925 raft_consensus.cc:729] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f053fb6252874522a9d7946b743f7d38, State: Initialized, Role: FOLLOWER
I20260619 18:58:05.054160 925 consensus_queue.cc:260] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f053fb6252874522a9d7946b743f7d38" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44307 } } peers { permanent_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39627 } } peers { permanent_uuid: "232f796689934f76a9ee047e5c0fa28e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 37841 } }
I20260619 18:58:05.054966 927 raft_consensus.cc:348] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f053fb6252874522a9d7946b743f7d38" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44307 } } peers { permanent_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39627 } } peers { permanent_uuid: "232f796689934f76a9ee047e5c0fa28e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 37841 } }
I20260619 18:58:05.055580 771 heartbeater.cc:503] Master 127.15.91.62:39011 was elected leader, sending a full tablet report...
I20260619 18:58:05.055600 927 raft_consensus.cc:374] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:05.055701 927 raft_consensus.cc:729] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232f796689934f76a9ee047e5c0fa28e, State: Initialized, Role: FOLLOWER
I20260619 18:58:05.056313 925 ts_tablet_manager.cc:1428] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260619 18:58:05.056543 927 consensus_queue.cc:260] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f053fb6252874522a9d7946b743f7d38" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44307 } } peers { permanent_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39627 } } peers { permanent_uuid: "232f796689934f76a9ee047e5c0fa28e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 37841 } }
I20260619 18:58:05.058960 927 ts_tablet_manager.cc:1428] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260619 18:58:05.122308 907 tablet.cc:2369] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:58:05.179077 932 raft_consensus.cc:482] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:58:05.179610 932 raft_consensus.cc:504] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f053fb6252874522a9d7946b743f7d38" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44307 } } peers { permanent_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39627 } } peers { permanent_uuid: "232f796689934f76a9ee047e5c0fa28e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 37841 } }
I20260619 18:58:05.182014 932 leader_election.cc:288] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bd3dc715c20460c8bd9300d829c4a7f (127.15.91.3:39627), 232f796689934f76a9ee047e5c0fa28e (127.15.91.1:37841)
W20260619 18:58:05.183969 638 tablet.cc:2369] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:58:05.191627 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:05.182782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:44941 (local address 127.15.91.1:37841)
0619 18:58:05.183049 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:05.183073 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:05.183950 (+ 877us) server_negotiation.cc:407] Connection header received
0619 18:58:05.184262 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:05.184268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:05.184357 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:05.184427 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:05.185206 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.186130 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.189172 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.190236 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.190393 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:05.190491 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:05.191153 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 18:58:05.191328 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:58:05.191676 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:05.182509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:37841 (local address 127.15.91.2:44941)
0619 18:58:05.183798 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:05.183817 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:05.184046 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:05.184618 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:05.184622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:05.184646 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:05.185021 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:05.185032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.186381 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.186390 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:05.188864 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:05.188889 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.190619 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.190627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:05.190906 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:05.190951 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 18:58:05.191089 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 18:58:05.191276 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1110,"thread_start_us":612,"threads_started":1}
I20260619 18:58:05.192269 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:05.182585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:52601 (local address 127.15.91.3:39627)
0619 18:58:05.182883 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:05.182892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:05.184021 (+ 1129us) server_negotiation.cc:407] Connection header received
0619 18:58:05.184362 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:05.184368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:05.184457 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:05.184529 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:05.185709 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.187098 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.189586 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.190978 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.191187 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:05.191300 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:05.191726 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 18:58:05.191926 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 18:58:05.192310 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:05.182564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:39627 (local address 127.15.91.2:52601)
0619 18:58:05.183886 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:05.183912 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:05.184167 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:05.184770 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:05.184779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:05.184892 (+ 113us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:05.185391 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:05.185405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.187543 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.187549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:05.189315 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:05.189334 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.191276 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.191284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:05.191502 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:05.191538 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:58:05.191675 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 18:58:05.191863 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":1,"spinlock_wait_cycles":29568,"thread_start_us":896,"threads_started":1}
I20260619 18:58:05.193101 592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1753caf4c8cf4f3ba7d1d4675d7db750" candidate_uuid: "f053fb6252874522a9d7946b743f7d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232f796689934f76a9ee047e5c0fa28e" is_pre_election: true
I20260619 18:58:05.193382 861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1753caf4c8cf4f3ba7d1d4675d7db750" candidate_uuid: "f053fb6252874522a9d7946b743f7d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" is_pre_election: true
I20260619 18:58:05.193761 592 raft_consensus.cc:2454] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f053fb6252874522a9d7946b743f7d38 in term 0.
I20260619 18:58:05.193996 861 raft_consensus.cc:2454] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f053fb6252874522a9d7946b743f7d38 in term 0.
I20260619 18:58:05.194703 662 leader_election.cc:302] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 232f796689934f76a9ee047e5c0fa28e, 6bd3dc715c20460c8bd9300d829c4a7f, f053fb6252874522a9d7946b743f7d38; no voters:
I20260619 18:58:05.195282 932 raft_consensus.cc:2781] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:58:05.195454 932 raft_consensus.cc:482] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:58:05.195583 932 raft_consensus.cc:3037] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:05.198237 932 raft_consensus.cc:504] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f053fb6252874522a9d7946b743f7d38" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44307 } } peers { permanent_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39627 } } peers { permanent_uuid: "232f796689934f76a9ee047e5c0fa28e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 37841 } }
I20260619 18:58:05.199116 932 leader_election.cc:288] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [CANDIDATE]: Term 1 election: Requested vote from peers 6bd3dc715c20460c8bd9300d829c4a7f (127.15.91.3:39627), 232f796689934f76a9ee047e5c0fa28e (127.15.91.1:37841)
I20260619 18:58:05.199882 861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1753caf4c8cf4f3ba7d1d4675d7db750" candidate_uuid: "f053fb6252874522a9d7946b743f7d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd3dc715c20460c8bd9300d829c4a7f"
I20260619 18:58:05.199883 592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1753caf4c8cf4f3ba7d1d4675d7db750" candidate_uuid: "f053fb6252874522a9d7946b743f7d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232f796689934f76a9ee047e5c0fa28e"
I20260619 18:58:05.200170 592 raft_consensus.cc:3037] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:05.200160 861 raft_consensus.cc:3037] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:05.203858 861 raft_consensus.cc:2454] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f053fb6252874522a9d7946b743f7d38 in term 1.
I20260619 18:58:05.203858 592 raft_consensus.cc:2454] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f053fb6252874522a9d7946b743f7d38 in term 1.
I20260619 18:58:05.204530 662 leader_election.cc:302] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 232f796689934f76a9ee047e5c0fa28e, 6bd3dc715c20460c8bd9300d829c4a7f, f053fb6252874522a9d7946b743f7d38; no voters:
I20260619 18:58:05.204946 932 raft_consensus.cc:2781] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:05.205520 932 raft_consensus.cc:686] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 1 LEADER]: Becoming Leader. State: Replica: f053fb6252874522a9d7946b743f7d38, State: Running, Role: LEADER
I20260619 18:58:05.206099 932 consensus_queue.cc:237] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f053fb6252874522a9d7946b743f7d38" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44307 } } peers { permanent_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39627 } } peers { permanent_uuid: "232f796689934f76a9ee047e5c0fa28e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 37841 } }
I20260619 18:58:05.211982 447 catalog_manager.cc:5495] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 reported cstate change: term changed from 0 to 1, leader changed from <none> to f053fb6252874522a9d7946b743f7d38 (127.15.91.2). New cstate: current_term: 1 leader_uuid: "f053fb6252874522a9d7946b743f7d38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f053fb6252874522a9d7946b743f7d38" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 44307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "232f796689934f76a9ee047e5c0fa28e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 37841 } health_report { overall_health: UNKNOWN } } }
I20260619 18:58:05.240036 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:05.232850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.15.91.2:44307)
0619 18:58:05.233206 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:05.233214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:05.233354 (+ 140us) server_negotiation.cc:407] Connection header received
0619 18:58:05.233654 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:05.233658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:05.233761 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:05.233944 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:58:05.234389 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.235621 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.237364 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.237810 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.237981 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:05.238596 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:58:05.239258 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:58:05.239438 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:05.239609 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 18:58:05.239708 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260619 18:58:05.250368 592 raft_consensus.cc:1261] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e [term 1 FOLLOWER]: Refusing update from remote peer f053fb6252874522a9d7946b743f7d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:05.250368 861 raft_consensus.cc:1261] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f [term 1 FOLLOWER]: Refusing update from remote peer f053fb6252874522a9d7946b743f7d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:05.251968 936 consensus_queue.cc:1035] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "232f796689934f76a9ee047e5c0fa28e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 37841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:05.252429 932 consensus_queue.cc:1035] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd3dc715c20460c8bd9300d829c4a7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 39627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:05.266026 939 mvcc.cc:205] Tried to move back new op lower bound from 7298643907571376128 to 7298643907409420288. Current Snapshot: MvccSnapshot[applied={T|T < 7298643907571376128}]
I20260619 18:58:05.287907 447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:05.288287 447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:05.293365 447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06e085dad19c430cbdfdedbae47451e4: Sending DeleteTablet for 3 replicas of tablet 1753caf4c8cf4f3ba7d1d4675d7db750
I20260619 18:58:05.294900 841 tablet_service.cc:1514] Processing DeleteTablet for tablet 1753caf4c8cf4f3ba7d1d4675d7db750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:05 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:60264
I20260619 18:58:05.294900 706 tablet_service.cc:1514] Processing DeleteTablet for tablet 1753caf4c8cf4f3ba7d1d4675d7db750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:05 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:40846
I20260619 18:58:05.299113 572 tablet_service.cc:1514] Processing DeleteTablet for tablet 1753caf4c8cf4f3ba7d1d4675d7db750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:05 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:44238
I20260619 18:58:05.301546 950 tablet_replica.cc:330] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: stopping tablet replica
I20260619 18:58:05.302366 950 raft_consensus.cc:2229] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:58:05.303364 950 raft_consensus.cc:2258] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:05.309012 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 509
I20260619 18:58:05.309232 951 tablet_replica.cc:330] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38: stopping tablet replica
I20260619 18:58:05.309779 951 raft_consensus.cc:2229] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:58:05.310624 951 raft_consensus.cc:2258] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:05.313625 950 ts_tablet_manager.cc:1905] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:05.314670 952 tablet_replica.cc:330] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: stopping tablet replica
I20260619 18:58:05.315421 952 raft_consensus.cc:2229] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:58:05.316401 952 raft_consensus.cc:2258] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:05.319687 951 ts_tablet_manager.cc:1905] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:05.321575 952 ts_tablet_manager.cc:1905] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:05.324891 950 ts_tablet_manager.cc:1918] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:05.325199 950 log.cc:1196] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/1753caf4c8cf4f3ba7d1d4675d7db750
I20260619 18:58:05.325834 950 ts_tablet_manager.cc:1939] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 6bd3dc715c20460c8bd9300d829c4a7f: Deleting consensus metadata
I20260619 18:58:05.328235 435 catalog_manager.cc:4841] TS 6bd3dc715c20460c8bd9300d829c4a7f (127.15.91.3:39627): tablet 1753caf4c8cf4f3ba7d1d4675d7db750 (table test-table [id=d8245e47768346f6a850c102ebb72d5c]) successfully deleted
I20260619 18:58:05.329914 951 ts_tablet_manager.cc:1918] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:05.330168 951 log.cc:1196] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/1753caf4c8cf4f3ba7d1d4675d7db750
I20260619 18:58:05.330964 951 ts_tablet_manager.cc:1939] T 1753caf4c8cf4f3ba7d1d4675d7db750 P f053fb6252874522a9d7946b743f7d38: Deleting consensus metadata
I20260619 18:58:05.333233 433 catalog_manager.cc:4841] TS f053fb6252874522a9d7946b743f7d38 (127.15.91.2:44307): tablet 1753caf4c8cf4f3ba7d1d4675d7db750 (table test-table [id=d8245e47768346f6a850c102ebb72d5c]) successfully deleted
I20260619 18:58:05.334041 952 ts_tablet_manager.cc:1918] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:05.334276 952 log.cc:1196] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/1753caf4c8cf4f3ba7d1d4675d7db750
I20260619 18:58:05.334846 952 ts_tablet_manager.cc:1939] T 1753caf4c8cf4f3ba7d1d4675d7db750 P 232f796689934f76a9ee047e5c0fa28e: Deleting consensus metadata
Jun 19 18:58:05 dist-test-slave-dm3f krb5kdc[403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:58:05.336994 432 catalog_manager.cc:4841] TS 232f796689934f76a9ee047e5c0fa28e (127.15.91.1:37841): tablet 1753caf4c8cf4f3ba7d1d4675d7db750 (table test-table [id=d8245e47768346f6a850c102ebb72d5c]) successfully deleted
I20260619 18:58:05.346242 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:05.310792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.15.91.1:37841)
0619 18:58:05.310991 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:05.310998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:05.317696 (+ 6698us) server_negotiation.cc:407] Connection header received
0619 18:58:05.317791 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:05.317797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:05.317893 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:05.318031 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:05.322434 (+ 4403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.323624 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.330400 (+ 6776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.331208 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.331359 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:05.337732 (+ 6373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:05.337750 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:05.337770 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:05.337833 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:05.342056 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:05.343322 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:05.343338 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:05.343344 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:05.343663 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:05.344216 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:05.344222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:05.344228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:05.344792 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:05.345134 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:05.345526 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 18:58:05.345809 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:58:05.347223 632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:05.484335 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 509
I20260619 18:58:05.504066 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 643
Jun 19 18:58:05 dist-test-slave-dm3f krb5kdc[403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:58:05.523409 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:05.504956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.15.91.2:44307)
0619 18:58:05.505144 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:05.505152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:05.505190 (+ 38us) server_negotiation.cc:407] Connection header received
0619 18:58:05.505421 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:05.505431 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:05.505508 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:05.505644 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:05.507092 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.508003 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.509608 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.510083 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.510242 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:05.515405 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:05.515422 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:05.515439 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:05.515495 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:05.519921 (+ 4426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:05.520851 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:05.520860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:05.520863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:05.521042 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:05.521496 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:05.521501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:05.521506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:05.521956 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:05.522367 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:05.522701 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 18:58:05.522996 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:58:05.524396 766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:05.643510 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 643
I20260619 18:58:05.660254 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 777
Jun 19 18:58:05 dist-test-slave-dm3f krb5kdc[403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:58:05.677780 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:05.661063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.15.91.3:39627)
0619 18:58:05.661300 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:05.661308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:05.661422 (+ 114us) server_negotiation.cc:407] Connection header received
0619 18:58:05.661644 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:05.661650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:05.661737 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:05.661874 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:05.663370 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.664118 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.665484 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:05.665901 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:05.666052 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:05.671025 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:05.671044 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:05.671062 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:05.671108 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:05.674040 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:05.675317 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:05.675326 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:05.675328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:05.675516 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:05.676090 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:05.676094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:05.676096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:05.676538 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:05.676815 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:05.677277 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 18:58:05.677480 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:58:05.678560 901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:05.771064 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 777
I20260619 18:58:05.789194 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 416
I20260619 18:58:05.790426 478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:05.907527 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 416
2026-06-19T18:58:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4851 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T18:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:05Z Disabled control of system clock
I20260619 18:58:05.996500 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:37175
--webserver_interface=127.15.91.62
--webserver_port=0
--builtin_ntp_servers=127.15.91.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:37175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:58:06.373397 961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:06.373739 961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:06.373826 961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:06.381721 961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:06.381855 961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:06.381896 961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:06.381955 961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:06.382007 961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:06.392063 961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:37175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:37175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:06.393566 961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:06.395566 961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:06.409264 967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:06.409300 966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:06.409327 969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:06.410182 961 server_base.cc:1034] running on GCE node
I20260619 18:58:06.411211 961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:06.413039 961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:06.414376 961 hybrid_clock.cc:648] HybridClock initialized: now 1781895486414321 us; error 48 us; skew 500 ppm
I20260619 18:58:06.414947 961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:06.423524 961 webserver.cc:460] Webserver started at http://127.15.91.62:32817/ using document root <none> and password file <none>
I20260619 18:58:06.424528 961 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:06.424734 961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:06.425136 961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:06.429500 961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "de3eaede0186457681f991f1f530a084"
format_stamp: "Formatted at 2026-06-19 18:58:06 on dist-test-slave-dm3f"
server_key: "ffb51dcdd561bb64d20e403593f03f4b"
server_key_iv: "903e0d0f0368d06099688107dba2b6cf"
server_key_version: "encryptionkey@0"
I20260619 18:58:06.431028 961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "de3eaede0186457681f991f1f530a084"
format_stamp: "Formatted at 2026-06-19 18:58:06 on dist-test-slave-dm3f"
server_key: "ffb51dcdd561bb64d20e403593f03f4b"
server_key_iv: "903e0d0f0368d06099688107dba2b6cf"
server_key_version: "encryptionkey@0"
I20260619 18:58:06.440893 961 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.006s
I20260619 18:58:06.446061 975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:06.448160 961 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:06.448436 961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "de3eaede0186457681f991f1f530a084"
format_stamp: "Formatted at 2026-06-19 18:58:06 on dist-test-slave-dm3f"
server_key: "ffb51dcdd561bb64d20e403593f03f4b"
server_key_iv: "903e0d0f0368d06099688107dba2b6cf"
server_key_version: "encryptionkey@0"
I20260619 18:58:06.448734 961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:06.477103 961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:06.478148 961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:06.478515 961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:06.501595 961 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:37175
I20260619 18:58:06.501607 1026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:37175 every 8 connection(s)
I20260619 18:58:06.503305 961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:06.505595 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 961
I20260619 18:58:06.505796 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:06.506424 15724 external_mini_cluster.cc:1428] Setting key d59f37e7ff4b914ef8246a1fb9da1561
I20260619 18:58:06.509199 1027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:06.524303 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:06.508796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.15.91.62:37175)
0619 18:58:06.510076 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:06.510088 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:06.510140 (+ 52us) server_negotiation.cc:407] Connection header received
0619 18:58:06.512627 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:06.512658 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:06.513507 (+ 849us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:06.514038 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:06.514817 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:06.516561 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:06.518192 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:06.519097 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:06.519355 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:06.521056 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:06.521066 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:06.521089 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:06.521806 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:06.521908 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:06.521940 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:06.522063 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:06.522679 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 18:58:06.523159 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":511,"threads_started":1}
I20260619 18:58:06.524435 1027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084: Bootstrap starting.
I20260619 18:58:06.528271 1027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:06.529515 1027 log.cc:824] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:06.535591 1027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084: No bootstrap required, opened a new log
I20260619 18:58:06.543843 1027 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3eaede0186457681f991f1f530a084" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37175 } }
I20260619 18:58:06.544313 1027 raft_consensus.cc:374] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:06.544459 1027 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3eaede0186457681f991f1f530a084, State: Initialized, Role: FOLLOWER
I20260619 18:58:06.545253 1027 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3eaede0186457681f991f1f530a084" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37175 } }
I20260619 18:58:06.545506 1027 raft_consensus.cc:388] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:06.545681 1027 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:06.545855 1027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:06.548480 1027 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3eaede0186457681f991f1f530a084" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37175 } }
I20260619 18:58:06.549161 1027 leader_election.cc:302] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de3eaede0186457681f991f1f530a084; no voters:
I20260619 18:58:06.549862 1027 leader_election.cc:288] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:06.550158 1032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:06.551007 1032 raft_consensus.cc:686] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [term 1 LEADER]: Becoming Leader. State: Replica: de3eaede0186457681f991f1f530a084, State: Running, Role: LEADER
I20260619 18:58:06.551667 1032 consensus_queue.cc:237] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3eaede0186457681f991f1f530a084" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37175 } }
I20260619 18:58:06.552520 1027 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:06.555128 1033 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de3eaede0186457681f991f1f530a084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3eaede0186457681f991f1f530a084" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37175 } } }
I20260619 18:58:06.555089 1034 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de3eaede0186457681f991f1f530a084. Latest consensus state: current_term: 1 leader_uuid: "de3eaede0186457681f991f1f530a084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3eaede0186457681f991f1f530a084" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 37175 } } }
I20260619 18:58:06.555768 1034 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:06.555790 1033 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:06.557983 1038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:06.563866 1038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:06.574577 1038 catalog_manager.cc:1347] Generated new cluster ID: 6eed9f3c32a74444b08292eb092c5d5b
I20260619 18:58:06.574759 1038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:06.610313 1038 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:06.611889 1038 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:06.622568 1038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084: Generated new TSK 0
I20260619 18:58:06.623661 1038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:06.645334 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37175
--builtin_ntp_servers=127.15.91.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:58:07.035917 1051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:07.036258 1051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:07.036355 1051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:07.044421 1051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:07.044644 1051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:07.044866 1051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:58:07.055051 1051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:07.056478 1051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:07.058626 1051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:07.073513 1056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:07.073414 1059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:07.073412 1057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:07.074499 1051 server_base.cc:1034] running on GCE node
I20260619 18:58:07.075313 1051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:07.076519 1051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:07.077844 1051 hybrid_clock.cc:648] HybridClock initialized: now 1781895487077780 us; error 70 us; skew 500 ppm
I20260619 18:58:07.078287 1051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:07.081190 1051 webserver.cc:460] Webserver started at http://127.15.91.1:39545/ using document root <none> and password file <none>
I20260619 18:58:07.082120 1051 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:07.082296 1051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:07.082675 1051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:07.086050 1051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "aad6b3c7a92c42238cad7da991e62724"
format_stamp: "Formatted at 2026-06-19 18:58:07 on dist-test-slave-dm3f"
server_key: "94eae7ec13fbbb5b711923718771f164"
server_key_iv: "cf181c29b5e9bfb611a95fefaaf39bea"
server_key_version: "encryptionkey@0"
I20260619 18:58:07.087150 1051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "aad6b3c7a92c42238cad7da991e62724"
format_stamp: "Formatted at 2026-06-19 18:58:07 on dist-test-slave-dm3f"
server_key: "94eae7ec13fbbb5b711923718771f164"
server_key_iv: "cf181c29b5e9bfb611a95fefaaf39bea"
server_key_version: "encryptionkey@0"
I20260619 18:58:07.093667 1051 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:58:07.098196 1065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:07.099983 1051 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:07.100246 1051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "aad6b3c7a92c42238cad7da991e62724"
format_stamp: "Formatted at 2026-06-19 18:58:07 on dist-test-slave-dm3f"
server_key: "94eae7ec13fbbb5b711923718771f164"
server_key_iv: "cf181c29b5e9bfb611a95fefaaf39bea"
server_key_version: "encryptionkey@0"
I20260619 18:58:07.100528 1051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:07.131079 1051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:07.132167 1051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:07.132515 1051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:07.133699 1051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:07.135711 1051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:07.135880 1051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:07.136049 1051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:07.136147 1051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:07.174782 1051 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:32973
I20260619 18:58:07.174903 1177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:32973 every 8 connection(s)
I20260619 18:58:07.176313 1051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:58:07.178074 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 1051
I20260619 18:58:07.178274 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:58:07.178926 15724 external_mini_cluster.cc:1428] Setting key bec0cdc639d191715b33095bad5bdb4e
I20260619 18:58:07.182557 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37175
--builtin_ntp_servers=127.15.91.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:58:07.195214 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:07.179559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:39399 (local address 127.15.91.62:37175)
0619 18:58:07.180152 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:07.180165 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:07.182033 (+ 1868us) server_negotiation.cc:407] Connection header received
0619 18:58:07.185350 (+ 3317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:07.185355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:07.185457 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:07.185569 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:07.187372 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:07.188140 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:07.191146 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:07.191594 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:07.191753 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:07.193939 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:07.193950 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:07.193959 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:07.193991 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:07.194052 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:07.194072 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:07.194202 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:07.194546 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 18:58:07.194701 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":338,"threads_started":1}
I20260619 18:58:07.196313 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:07.180364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:37175 (local address 127.15.91.1:39399)
0619 18:58:07.181800 (+ 1436us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:07.181871 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:07.184890 (+ 3019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:07.185926 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:07.185942 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:07.186095 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:07.187092 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:07.187133 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:07.188372 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:07.188382 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:07.190863 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:07.190895 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:07.191949 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:07.191958 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:07.192367 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:07.193438 (+ 1071us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:58:07.193442 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:07.193507 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:58:07.193611 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:07.194256 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:07.194267 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:07.194276 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:58:07.194496 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 18:58:07.194980 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":442,"threads_started":1}
I20260619 18:58:07.198397 1178 heartbeater.cc:347] Connected to a master server at 127.15.91.62:37175
I20260619 18:58:07.198843 1178 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:07.199960 1178 heartbeater.cc:511] Master 127.15.91.62:37175 requested a full tablet report, sending...
I20260619 18:58:07.203006 992 ts_manager.cc:194] Registered new tserver with Master: aad6b3c7a92c42238cad7da991e62724 (127.15.91.1:32973)
I20260619 18:58:07.204841 992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.1:39399
W20260619 18:58:07.556842 1182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:07.557176 1182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:07.557330 1182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:07.565147 1182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:07.565279 1182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:07.565475 1182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:58:07.575240 1182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:07.576659 1182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:07.578627 1182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:07.593392 1188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:07.593520 1187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:07.593510 1190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:07.594738 1182 server_base.cc:1034] running on GCE node
I20260619 18:58:07.595491 1182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:07.596657 1182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:07.597978 1182 hybrid_clock.cc:648] HybridClock initialized: now 1781895487597904 us; error 74 us; skew 500 ppm
I20260619 18:58:07.598340 1182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:07.601027 1182 webserver.cc:460] Webserver started at http://127.15.91.2:40237/ using document root <none> and password file <none>
I20260619 18:58:07.601807 1182 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:07.601909 1182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:07.602191 1182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:07.605728 1182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "ad778921515843a6a9e7548381a4a924"
format_stamp: "Formatted at 2026-06-19 18:58:07 on dist-test-slave-dm3f"
server_key: "6ef155429e04576b6c054e228b2cded3"
server_key_iv: "e16b6433a1261dc05dc2d26a38e56332"
server_key_version: "encryptionkey@0"
I20260619 18:58:07.606657 1182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "ad778921515843a6a9e7548381a4a924"
format_stamp: "Formatted at 2026-06-19 18:58:07 on dist-test-slave-dm3f"
server_key: "6ef155429e04576b6c054e228b2cded3"
server_key_iv: "e16b6433a1261dc05dc2d26a38e56332"
server_key_version: "encryptionkey@0"
I20260619 18:58:07.612838 1182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 18:58:07.617763 1196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:07.619482 1182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:58:07.619663 1182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "ad778921515843a6a9e7548381a4a924"
format_stamp: "Formatted at 2026-06-19 18:58:07 on dist-test-slave-dm3f"
server_key: "6ef155429e04576b6c054e228b2cded3"
server_key_iv: "e16b6433a1261dc05dc2d26a38e56332"
server_key_version: "encryptionkey@0"
I20260619 18:58:07.619889 1182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:07.643492 1182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:07.644389 1182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:07.644665 1182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:07.645742 1182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:07.647482 1182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:07.647624 1182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:07.647764 1182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:07.647836 1182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:07.685527 1182 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:35701
I20260619 18:58:07.685734 1308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:35701 every 8 connection(s)
I20260619 18:58:07.687099 1182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:58:07.688772 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 1182
I20260619 18:58:07.688964 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:58:07.689435 15724 external_mini_cluster.cc:1428] Setting key 44db7f68b42e7d41462f6408a106f4f9
I20260619 18:58:07.692966 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37175
--builtin_ntp_servers=127.15.91.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:58:07.703801 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:07.690192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:39325 (local address 127.15.91.62:37175)
0619 18:58:07.690380 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:07.690385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:07.692405 (+ 2020us) server_negotiation.cc:407] Connection header received
0619 18:58:07.694558 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:07.694564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:07.694677 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:07.694835 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:07.696139 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:07.696944 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:07.700099 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:07.700853 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:07.701057 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:07.702797 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:07.702807 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:07.702816 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:07.702870 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:07.702906 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:07.702921 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:07.703035 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:07.703314 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 18:58:07.703437 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:58:07.705246 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:07.690996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:37175 (local address 127.15.91.2:39325)
0619 18:58:07.692237 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:07.692279 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:07.694200 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:07.695120 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:07.695130 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:07.695255 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:07.695918 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:07.695936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:07.697223 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:07.697233 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:07.699797 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:07.699830 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:07.701195 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:07.701202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:07.701467 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:07.702307 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:58:07.702311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:07.702363 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:58:07.702471 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:07.703039 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:07.703055 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:07.703065 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:58:07.703299 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 18:58:07.703808 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":377,"threads_started":1}
I20260619 18:58:07.707221 1309 heartbeater.cc:347] Connected to a master server at 127.15.91.62:37175
I20260619 18:58:07.707616 1309 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:07.708549 1309 heartbeater.cc:511] Master 127.15.91.62:37175 requested a full tablet report, sending...
I20260619 18:58:07.710362 992 ts_manager.cc:194] Registered new tserver with Master: ad778921515843a6a9e7548381a4a924 (127.15.91.2:35701)
I20260619 18:58:07.711301 992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.2:39325
W20260619 18:58:08.061637 1312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:08.062019 1312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:08.062187 1312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:08.069926 1312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:08.070113 1312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:08.070319 1312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:58:08.080165 1312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:37175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:08.081678 1312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:08.083825 1312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:08.099040 1317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:08.098994 1320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:08.098961 1318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:08.100164 1312 server_base.cc:1034] running on GCE node
I20260619 18:58:08.100947 1312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:08.102154 1312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:08.103427 1312 hybrid_clock.cc:648] HybridClock initialized: now 1781895488103358 us; error 69 us; skew 500 ppm
I20260619 18:58:08.103826 1312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:08.106845 1312 webserver.cc:460] Webserver started at http://127.15.91.3:36673/ using document root <none> and password file <none>
I20260619 18:58:08.107715 1312 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:08.107879 1312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:08.108245 1312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:08.111555 1312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "1d9454b4904942bf9a577bb283488ac9"
format_stamp: "Formatted at 2026-06-19 18:58:08 on dist-test-slave-dm3f"
server_key: "da919ae5ff82a15e701e4910f83e5eb5"
server_key_iv: "9c02f3641b228ffa9bcd6130b278042a"
server_key_version: "encryptionkey@0"
I20260619 18:58:08.112557 1312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "1d9454b4904942bf9a577bb283488ac9"
format_stamp: "Formatted at 2026-06-19 18:58:08 on dist-test-slave-dm3f"
server_key: "da919ae5ff82a15e701e4910f83e5eb5"
server_key_iv: "9c02f3641b228ffa9bcd6130b278042a"
server_key_version: "encryptionkey@0"
I20260619 18:58:08.119057 1312 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260619 18:58:08.123263 1326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:08.125201 1312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:58:08.125453 1312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "1d9454b4904942bf9a577bb283488ac9"
format_stamp: "Formatted at 2026-06-19 18:58:08 on dist-test-slave-dm3f"
server_key: "da919ae5ff82a15e701e4910f83e5eb5"
server_key_iv: "9c02f3641b228ffa9bcd6130b278042a"
server_key_version: "encryptionkey@0"
I20260619 18:58:08.125720 1312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:08.144418 1312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:08.145408 1312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:08.145744 1312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:08.146981 1312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:08.148923 1312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:08.149096 1312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:08.149255 1312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:08.149356 1312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:08.187693 1312 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:44167
I20260619 18:58:08.187748 1438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:44167 every 8 connection(s)
I20260619 18:58:08.189332 1312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:58:08.193097 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 1312
I20260619 18:58:08.193310 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:58:08.193781 15724 external_mini_cluster.cc:1428] Setting key f0bbb0cfd5a88b745a34633ad214749f
I20260619 18:58:08.204457 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.192340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:55771 (local address 127.15.91.62:37175)
0619 18:58:08.192564 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:08.192568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:08.194518 (+ 1950us) server_negotiation.cc:407] Connection header received
0619 18:58:08.196789 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:08.196794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:08.196900 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:08.197064 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:08.198386 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.199256 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.201126 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.201550 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.201708 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.203419 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:08.203425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:08.203428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:08.203452 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:08.203500 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:08.203513 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:08.203618 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:08.203940 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 18:58:08.204067 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:58:08.205757 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.192906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:37175 (local address 127.15.91.3:55771)
0619 18:58:08.194266 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:08.194327 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:08.196460 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:08.197344 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:08.197355 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:08.197464 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:08.198151 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.198172 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.199460 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.199465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.200842 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.200859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.201830 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.201838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.202120 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.202983 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:58:08.202986 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:08.203036 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:58:08.203145 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:08.203638 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:08.203652 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:08.203666 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 18:58:08.203926 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 18:58:08.204486 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":501,"threads_started":1}
I20260619 18:58:08.207654 1439 heartbeater.cc:347] Connected to a master server at 127.15.91.62:37175
I20260619 18:58:08.208055 1439 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:08.209049 1439 heartbeater.cc:511] Master 127.15.91.62:37175 requested a full tablet report, sending...
I20260619 18:58:08.209787 1178 heartbeater.cc:503] Master 127.15.91.62:37175 was elected leader, sending a full tablet report...
I20260619 18:58:08.210922 992 ts_manager.cc:194] Registered new tserver with Master: 1d9454b4904942bf9a577bb283488ac9 (127.15.91.3:44167)
I20260619 18:58:08.211894 992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.3:55771
I20260619 18:58:08.222692 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:58:08.234522 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.227865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.15.91.62:37175)
0619 18:58:08.228156 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:08.228166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:08.228520 (+ 354us) server_negotiation.cc:407] Connection header received
0619 18:58:08.228778 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:08.228782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:08.228854 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:08.228969 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:08.229530 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.230307 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.231913 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.232360 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.232522 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.233723 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:08.233727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:08.233731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:08.233754 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:08.233789 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:08.233802 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:08.233908 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:08.234068 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 18:58:08.234175 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:58:08.239301 992 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:58:08.246591 992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:58:08.283026 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.261356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.15.91.3:44167)
0619 18:58:08.272255 (+ 10899us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:08.272267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:08.273263 (+ 996us) server_negotiation.cc:407] Connection header received
0619 18:58:08.273406 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:08.273415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:08.273716 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:08.274819 (+ 1103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:08.275160 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.276782 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.279907 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.281497 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.281664 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.281997 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:08.282340 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 18:58:08.282560 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10636,"thread_start_us":10919,"threads_started":1}
I20260619 18:58:08.283099 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.261153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:44167 (local address 127.0.0.1:47378)
0619 18:58:08.272691 (+ 11538us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:08.272748 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:08.273001 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:08.273993 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:08.274003 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:08.274048 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:08.274541 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.274563 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.276943 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.276951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.279615 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.279634 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.281753 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.281761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.282040 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.282095 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 18:58:08.282264 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 18:58:08.282540 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11176,"mutex_wait_us":1,"thread_start_us":12200,"threads_started":1}
I20260619 18:58:08.285225 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.261310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:32973 (local address 127.0.0.1:57930)
0619 18:58:08.273992 (+ 12682us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:08.274013 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:08.274155 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:08.276204 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:08.276210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:08.276243 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:08.276629 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.276640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.278800 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.278806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.281931 (+ 3125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.281949 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.284358 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.284366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.284567 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.284617 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 18:58:08.284720 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:58:08.284914 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12225,"mutex_wait_us":297,"spinlock_wait_cycles":1280,"thread_start_us":10853,"threads_started":1}
I20260619 18:58:08.286329 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.262000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57930 (local address 127.15.91.1:32973)
0619 18:58:08.272953 (+ 10953us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:08.272963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:08.274311 (+ 1348us) server_negotiation.cc:407] Connection header received
0619 18:58:08.275378 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:08.275390 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:08.275723 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:08.275935 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:08.276839 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.278539 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.282172 (+ 3633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.284095 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.284265 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.284590 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:08.285681 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0619 18:58:08.285918 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10758,"thread_start_us":10974,"threads_started":1}
I20260619 18:58:08.288460 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.274961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:35701 (local address 127.0.0.1:53016)
0619 18:58:08.278875 (+ 3914us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:08.278892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:08.279099 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:08.280195 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:08.280199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:08.280224 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:08.280680 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.280695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.282389 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.282394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.284684 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.284702 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.287567 (+ 2865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.287575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.287791 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.287820 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:58:08.287946 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:58:08.288115 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3774,"thread_start_us":3547,"threads_started":1}
I20260619 18:58:08.288581 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.273814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.15.91.2:35701)
0619 18:58:08.278279 (+ 4465us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:08.278290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:08.279024 (+ 734us) server_negotiation.cc:407] Connection header received
0619 18:58:08.279372 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:08.279383 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:08.279682 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:08.279935 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:08.280928 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.282216 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.284922 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.286496 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.287192 (+ 696us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.287463 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:08.288007 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 18:58:08.288208 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4236,"thread_start_us":4104,"threads_started":1}
I20260619 18:58:08.288316 1374 tablet_service.cc:1467] Processing CreateTablet for tablet 1edb631973da4c11a390e7c59043a51c (DEFAULT_TABLE table=test-table [id=7a9a45820abb4c5582355e328e586fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:08.290254 1374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edb631973da4c11a390e7c59043a51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:08.291163 1113 tablet_service.cc:1467] Processing CreateTablet for tablet 1edb631973da4c11a390e7c59043a51c (DEFAULT_TABLE table=test-table [id=7a9a45820abb4c5582355e328e586fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:08.292953 1113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edb631973da4c11a390e7c59043a51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:08.292644 1244 tablet_service.cc:1467] Processing CreateTablet for tablet 1edb631973da4c11a390e7c59043a51c (DEFAULT_TABLE table=test-table [id=7a9a45820abb4c5582355e328e586fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:08.294116 1244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edb631973da4c11a390e7c59043a51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:08.307274 1457 tablet_bootstrap.cc:492] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: Bootstrap starting.
I20260619 18:58:08.310781 1459 tablet_bootstrap.cc:492] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724: Bootstrap starting.
I20260619 18:58:08.311956 1457 tablet_bootstrap.cc:654] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:08.313202 1458 tablet_bootstrap.cc:492] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: Bootstrap starting.
I20260619 18:58:08.313690 1457 log.cc:824] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:08.315265 1459 tablet_bootstrap.cc:654] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:08.317170 1459 log.cc:824] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:08.318908 1458 tablet_bootstrap.cc:654] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:08.320502 1458 log.cc:824] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:08.320765 1457 tablet_bootstrap.cc:492] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: No bootstrap required, opened a new log
I20260619 18:58:08.321053 1459 tablet_bootstrap.cc:492] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724: No bootstrap required, opened a new log
I20260619 18:58:08.321123 1457 ts_tablet_manager.cc:1397] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260619 18:58:08.321425 1459 ts_tablet_manager.cc:1397] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260619 18:58:08.324903 1458 tablet_bootstrap.cc:492] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: No bootstrap required, opened a new log
I20260619 18:58:08.325273 1458 ts_tablet_manager.cc:1397] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260619 18:58:08.331426 1457 raft_consensus.cc:348] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9454b4904942bf9a577bb283488ac9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44167 } } peers { permanent_uuid: "aad6b3c7a92c42238cad7da991e62724" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 32973 } } peers { permanent_uuid: "ad778921515843a6a9e7548381a4a924" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 35701 } }
I20260619 18:58:08.331903 1457 raft_consensus.cc:374] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:08.331997 1457 raft_consensus.cc:729] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad778921515843a6a9e7548381a4a924, State: Initialized, Role: FOLLOWER
I20260619 18:58:08.332742 1457 consensus_queue.cc:260] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9454b4904942bf9a577bb283488ac9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44167 } } peers { permanent_uuid: "aad6b3c7a92c42238cad7da991e62724" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 32973 } } peers { permanent_uuid: "ad778921515843a6a9e7548381a4a924" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 35701 } }
I20260619 18:58:08.333283 1459 raft_consensus.cc:348] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9454b4904942bf9a577bb283488ac9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44167 } } peers { permanent_uuid: "aad6b3c7a92c42238cad7da991e62724" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 32973 } } peers { permanent_uuid: "ad778921515843a6a9e7548381a4a924" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 35701 } }
I20260619 18:58:08.333835 1459 raft_consensus.cc:374] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:08.333941 1459 raft_consensus.cc:729] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad6b3c7a92c42238cad7da991e62724, State: Initialized, Role: FOLLOWER
I20260619 18:58:08.334164 1309 heartbeater.cc:503] Master 127.15.91.62:37175 was elected leader, sending a full tablet report...
I20260619 18:58:08.334878 1457 ts_tablet_manager.cc:1428] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: Time spent starting tablet: real 0.014s user 0.009s sys 0.007s
I20260619 18:58:08.334640 1459 consensus_queue.cc:260] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9454b4904942bf9a577bb283488ac9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44167 } } peers { permanent_uuid: "aad6b3c7a92c42238cad7da991e62724" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 32973 } } peers { permanent_uuid: "ad778921515843a6a9e7548381a4a924" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 35701 } }
I20260619 18:58:08.336760 1459 ts_tablet_manager.cc:1428] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260619 18:58:08.337237 1458 raft_consensus.cc:348] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9454b4904942bf9a577bb283488ac9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44167 } } peers { permanent_uuid: "aad6b3c7a92c42238cad7da991e62724" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 32973 } } peers { permanent_uuid: "ad778921515843a6a9e7548381a4a924" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 35701 } }
I20260619 18:58:08.337760 1458 raft_consensus.cc:374] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:08.337859 1458 raft_consensus.cc:729] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9454b4904942bf9a577bb283488ac9, State: Initialized, Role: FOLLOWER
I20260619 18:58:08.338613 1458 consensus_queue.cc:260] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9454b4904942bf9a577bb283488ac9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44167 } } peers { permanent_uuid: "aad6b3c7a92c42238cad7da991e62724" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 32973 } } peers { permanent_uuid: "ad778921515843a6a9e7548381a4a924" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 35701 } }
I20260619 18:58:08.340598 1439 heartbeater.cc:503] Master 127.15.91.62:37175 was elected leader, sending a full tablet report...
I20260619 18:58:08.341122 1458 ts_tablet_manager.cc:1428] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
W20260619 18:58:08.429314 1179 tablet.cc:2369] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:58:08.439759 1310 tablet.cc:2369] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:58:08.441315 1440 tablet.cc:2369] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:58:08.488425 1465 raft_consensus.cc:482] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:58:08.488801 1465 raft_consensus.cc:504] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9454b4904942bf9a577bb283488ac9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44167 } } peers { permanent_uuid: "aad6b3c7a92c42238cad7da991e62724" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 32973 } } peers { permanent_uuid: "ad778921515843a6a9e7548381a4a924" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 35701 } }
I20260619 18:58:08.490653 1465 leader_election.cc:288] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad6b3c7a92c42238cad7da991e62724 (127.15.91.1:32973), ad778921515843a6a9e7548381a4a924 (127.15.91.2:35701)
I20260619 18:58:08.498361 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.491217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:41445 (local address 127.15.91.1:32973)
0619 18:58:08.491463 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:08.491470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:08.491582 (+ 112us) server_negotiation.cc:407] Connection header received
0619 18:58:08.491882 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:08.491890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:08.492014 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:08.492185 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:08.492894 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.493929 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.496090 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.497278 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.497395 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.497480 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:08.497940 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 18:58:08.498059 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260619 18:58:08.498396 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.491220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:32973 (local address 127.15.91.3:41445)
0619 18:58:08.491448 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:08.491470 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:08.491667 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:08.492383 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:08.492389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:08.492422 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:08.492675 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.492686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.494088 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.494094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.495856 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.495875 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.497484 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.497492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.497703 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.497743 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:58:08.497864 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 18:58:08.498028 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 18:58:08.499240 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.492009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:53497 (local address 127.15.91.2:35701)
0619 18:58:08.492255 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:08.492261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:08.492645 (+ 384us) server_negotiation.cc:407] Connection header received
0619 18:58:08.492810 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:08.492816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:08.492908 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:08.493008 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:08.493668 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.494630 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.496868 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.497998 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.498144 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.498250 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:08.498668 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 18:58:08.498925 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 18:58:08.499697 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.491685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:35701 (local address 127.15.91.3:53497)
0619 18:58:08.492507 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:08.492526 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:08.492661 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:08.493068 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:08.493073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:08.493100 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:08.493458 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.493470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.494886 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.494891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.496615 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:08.496634 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.498220 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.498228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:08.498441 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.498479 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 18:58:08.499219 (+ 740us) client_negotiation.cc:240] Negotiation successful
0619 18:58:08.499395 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":458,"threads_started":1}
I20260619 18:58:08.499776 1133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb631973da4c11a390e7c59043a51c" candidate_uuid: "1d9454b4904942bf9a577bb283488ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad6b3c7a92c42238cad7da991e62724" is_pre_election: true
I20260619 18:58:08.500442 1133 raft_consensus.cc:2454] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d9454b4904942bf9a577bb283488ac9 in term 0.
I20260619 18:58:08.500928 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb631973da4c11a390e7c59043a51c" candidate_uuid: "1d9454b4904942bf9a577bb283488ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad778921515843a6a9e7548381a4a924" is_pre_election: true
I20260619 18:58:08.501427 1327 leader_election.cc:302] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9454b4904942bf9a577bb283488ac9, aad6b3c7a92c42238cad7da991e62724; no voters:
I20260619 18:58:08.501638 1264 raft_consensus.cc:2454] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d9454b4904942bf9a577bb283488ac9 in term 0.
I20260619 18:58:08.501973 1465 raft_consensus.cc:2781] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:58:08.502173 1465 raft_consensus.cc:482] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:58:08.502278 1465 raft_consensus.cc:3037] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:08.505692 1465 raft_consensus.cc:504] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9454b4904942bf9a577bb283488ac9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44167 } } peers { permanent_uuid: "aad6b3c7a92c42238cad7da991e62724" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 32973 } } peers { permanent_uuid: "ad778921515843a6a9e7548381a4a924" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 35701 } }
I20260619 18:58:08.506494 1465 leader_election.cc:288] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [CANDIDATE]: Term 1 election: Requested vote from peers aad6b3c7a92c42238cad7da991e62724 (127.15.91.1:32973), ad778921515843a6a9e7548381a4a924 (127.15.91.2:35701)
I20260619 18:58:08.507166 1133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb631973da4c11a390e7c59043a51c" candidate_uuid: "1d9454b4904942bf9a577bb283488ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad6b3c7a92c42238cad7da991e62724"
I20260619 18:58:08.507256 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb631973da4c11a390e7c59043a51c" candidate_uuid: "1d9454b4904942bf9a577bb283488ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad778921515843a6a9e7548381a4a924"
I20260619 18:58:08.507463 1264 raft_consensus.cc:3037] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:08.507442 1133 raft_consensus.cc:3037] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:08.511013 1264 raft_consensus.cc:2454] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d9454b4904942bf9a577bb283488ac9 in term 1.
I20260619 18:58:08.511013 1133 raft_consensus.cc:2454] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d9454b4904942bf9a577bb283488ac9 in term 1.
I20260619 18:58:08.511579 1327 leader_election.cc:302] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9454b4904942bf9a577bb283488ac9, aad6b3c7a92c42238cad7da991e62724; no voters:
I20260619 18:58:08.511982 1465 raft_consensus.cc:2781] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:08.512504 1465 raft_consensus.cc:686] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 1 LEADER]: Becoming Leader. State: Replica: 1d9454b4904942bf9a577bb283488ac9, State: Running, Role: LEADER
I20260619 18:58:08.513113 1465 consensus_queue.cc:237] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9454b4904942bf9a577bb283488ac9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44167 } } peers { permanent_uuid: "aad6b3c7a92c42238cad7da991e62724" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 32973 } } peers { permanent_uuid: "ad778921515843a6a9e7548381a4a924" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 35701 } }
I20260619 18:58:08.519059 992 catalog_manager.cc:5495] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d9454b4904942bf9a577bb283488ac9 (127.15.91.3). New cstate: current_term: 1 leader_uuid: "1d9454b4904942bf9a577bb283488ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9454b4904942bf9a577bb283488ac9" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 44167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad6b3c7a92c42238cad7da991e62724" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad778921515843a6a9e7548381a4a924" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 35701 } health_report { overall_health: UNKNOWN } } }
I20260619 18:58:08.575565 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.569193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.15.91.3:44167)
0619 18:58:08.569460 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:08.569466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:08.569606 (+ 140us) server_negotiation.cc:407] Connection header received
0619 18:58:08.569899 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:08.569911 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:08.570011 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:08.570201 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:58:08.570612 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.571474 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.573049 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.573491 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.573661 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.574257 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:58:08.574862 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:58:08.575040 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:08.575185 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 18:58:08.575280 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 18:58:08.585078 1133 raft_consensus.cc:1261] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724 [term 1 FOLLOWER]: Refusing update from remote peer 1d9454b4904942bf9a577bb283488ac9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:08.585078 1264 raft_consensus.cc:1261] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924 [term 1 FOLLOWER]: Refusing update from remote peer 1d9454b4904942bf9a577bb283488ac9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:08.586326 1467 consensus_queue.cc:1035] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad6b3c7a92c42238cad7da991e62724" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:08.586789 1465 consensus_queue.cc:1035] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad778921515843a6a9e7548381a4a924" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 35701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:08.599884 1470 mvcc.cc:205] Tried to move back new op lower bound from 7298643921231941632 to 7298643920954564608. Current Snapshot: MvccSnapshot[applied={T|T < 7298643921231941632}]
I20260619 18:58:08.602324 1471 mvcc.cc:205] Tried to move back new op lower bound from 7298643921231941632 to 7298643920954564608. Current Snapshot: MvccSnapshot[applied={T|T < 7298643921231941632}]
I20260619 18:58:08.604835 1472 mvcc.cc:205] Tried to move back new op lower bound from 7298643921231941632 to 7298643920954564608. Current Snapshot: MvccSnapshot[applied={T|T < 7298643921231941632}]
I20260619 18:58:08.619715 991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:08.620056 991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:08.624473 991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de3eaede0186457681f991f1f530a084: Sending DeleteTablet for 3 replicas of tablet 1edb631973da4c11a390e7c59043a51c
I20260619 18:58:08.625995 1374 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edb631973da4c11a390e7c59043a51c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:08 UTC) from {username='slave'} at 127.0.0.1:47378
I20260619 18:58:08.626176 1113 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edb631973da4c11a390e7c59043a51c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:08 UTC) from {username='slave'} at 127.0.0.1:57930
I20260619 18:58:08.626467 1244 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edb631973da4c11a390e7c59043a51c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:08 UTC) from {username='slave'} at 127.0.0.1:53016
I20260619 18:58:08.629503 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 1051
I20260619 18:58:08.637733 1482 tablet_replica.cc:330] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: stopping tablet replica
I20260619 18:58:08.638476 1482 raft_consensus.cc:2229] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:58:08.639258 1482 raft_consensus.cc:2258] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:08.641878 1482 ts_tablet_manager.cc:1905] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:08.650063 1481 tablet_replica.cc:330] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724: stopping tablet replica
I20260619 18:58:08.650655 1481 raft_consensus.cc:2229] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:58:08.651324 1481 raft_consensus.cc:2258] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:08.652354 1480 tablet_replica.cc:330] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: stopping tablet replica
I20260619 18:58:08.652947 1480 raft_consensus.cc:2229] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:58:08.653885 1480 raft_consensus.cc:2258] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:08.654172 1481 ts_tablet_manager.cc:1905] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:08.656812 1480 ts_tablet_manager.cc:1905] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:08.659467 1482 ts_tablet_manager.cc:1918] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:08.659971 1482 log.cc:1196] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/1edb631973da4c11a390e7c59043a51c
I20260619 18:58:08.660751 1482 ts_tablet_manager.cc:1939] T 1edb631973da4c11a390e7c59043a51c P ad778921515843a6a9e7548381a4a924: Deleting consensus metadata
I20260619 18:58:08.662210 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.631619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.15.91.1:32973)
0619 18:58:08.631887 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:08.631896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:08.651473 (+ 19577us) server_negotiation.cc:407] Connection header received
0619 18:58:08.651829 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:08.651838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:08.652001 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:08.652122 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:08.652993 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.654168 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.656542 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.657277 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.657406 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.659349 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:08.659396 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:08.659426 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:08.660587 (+ 1161us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:08.660671 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:08.660711 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:08.660852 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:08.661626 (+ 774us) server_negotiation.cc:299] Negotiation successful
0619 18:58:08.661778 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:58:08.662990 976 catalog_manager.cc:4841] TS ad778921515843a6a9e7548381a4a924 (127.15.91.2:35701): tablet 1edb631973da4c11a390e7c59043a51c (table test-table [id=7a9a45820abb4c5582355e328e586fb6]) successfully deleted
I20260619 18:58:08.663269 1173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:08.667797 1480 ts_tablet_manager.cc:1918] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:08.668200 1480 log.cc:1196] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/1edb631973da4c11a390e7c59043a51c
I20260619 18:58:08.669042 1480 ts_tablet_manager.cc:1939] T 1edb631973da4c11a390e7c59043a51c P 1d9454b4904942bf9a577bb283488ac9: Deleting consensus metadata
I20260619 18:58:08.671574 979 catalog_manager.cc:4841] TS 1d9454b4904942bf9a577bb283488ac9 (127.15.91.3:44167): tablet 1edb631973da4c11a390e7c59043a51c (table test-table [id=7a9a45820abb4c5582355e328e586fb6]) successfully deleted
I20260619 18:58:08.803689 1481 ts_tablet_manager.cc:1918] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:08.803989 1481 log.cc:1196] T 1edb631973da4c11a390e7c59043a51c P aad6b3c7a92c42238cad7da991e62724: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/1edb631973da4c11a390e7c59043a51c
I20260619 18:58:08.804387 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 1051
W20260619 18:58:08.829758 976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.91.1:32973 (error 108)
I20260619 18:58:08.829906 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 1182
I20260619 18:58:08.832252 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.831216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:32973 (local address 127.0.0.1:57958)
0619 18:58:08.831843 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:08.832018 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:32973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":398,"threads_started":1}
W20260619 18:58:08.832679 976 catalog_manager.cc:4584] TS aad6b3c7a92c42238cad7da991e62724 (127.15.91.1:32973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1edb631973da4c11a390e7c59043a51c: Network error: Client connection negotiation failed: client connection to 127.15.91.1:32973: connect: Connection refused (error 111)
I20260619 18:58:08.839439 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.831299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.15.91.2:35701)
0619 18:58:08.831499 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:08.831505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:08.831967 (+ 462us) server_negotiation.cc:407] Connection header received
0619 18:58:08.832045 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:08.832050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:08.832163 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:08.832284 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:08.832796 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.833672 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.835958 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.836420 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.836547 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.837856 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:08.837865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:08.837882 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:08.838525 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:08.838586 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:08.838610 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:08.838861 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:08.838912 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 18:58:08.839020 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:58:08.840145 1304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:08.886638 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.886024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:32973 (local address 127.0.0.1:57960)
0619 18:58:08.886310 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:08.886434 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:32973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260619 18:58:08.888154 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.887608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:32973 (local address 127.0.0.1:57968)
0619 18:58:08.887838 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:08.887948 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:32973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 18:58:08.936123 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 1182
I20260619 18:58:08.949222 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.948415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:32973 (local address 127.0.0.1:57982)
0619 18:58:08.948731 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:08.948859 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:32973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 18:58:08.951390 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.950517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:32973 (local address 127.0.0.1:57994)
0619 18:58:08.950992 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:08.951135 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:32973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260619 18:58:08.958110 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 1312
I20260619 18:58:08.967415 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:08.959543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.15.91.3:44167)
0619 18:58:08.959773 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:08.959782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:08.959983 (+ 201us) server_negotiation.cc:407] Connection header received
0619 18:58:08.960259 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:08.960264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:08.960378 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:08.960522 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:08.960997 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.961848 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.963496 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:08.964088 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:08.964260 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:08.965543 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:08.965553 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:08.965569 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:08.966275 (+ 706us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:08.966336 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:08.966362 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:08.966544 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:08.966702 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 18:58:08.967102 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:58:08.968230 1434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:09.033015 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:09.032428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:32973 (local address 127.0.0.1:58000)
0619 18:58:09.032739 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:09.032845 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:32973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 18:58:09.034423 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:09.033879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:32973 (local address 127.0.0.1:58014)
0619 18:58:09.034144 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:09.034250 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:32973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 18:58:09.097755 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 1312
I20260619 18:58:09.118872 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 961
I20260619 18:58:09.120162 1022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:09.234481 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 961
I20260619 18:58:09.234550 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:09.233987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:32973 (local address 127.0.0.1:58018)
0619 18:58:09.234256 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:09.234379 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:32973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
2026-06-19T18:58:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3321 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:09 dist-test-slave-dm3f krb5kdc[1492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:09 dist-test-slave-dm3f krb5kdc[1492](info): set up 2 sockets
Jun 19 18:58:09 dist-test-slave-dm3f krb5kdc[1492](info): commencing operation
krb5kdc: starting...
W20260619 18:58:10.366945 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:10 dist-test-slave-dm3f krb5kdc[1492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:58:10.636054 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46257
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:11.007028 1506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:11.007454 1506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:11.007629 1506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:11.015745 1506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:11.015938 1506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:11.016040 1506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:11.016153 1506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:11.016235 1506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:11.027125 1506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:11.028679 1506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:11.030207 1506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:11.044809 1514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:11.044854 1512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:11.044888 1511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:11.045912 1506 server_base.cc:1034] running on GCE node
I20260619 18:58:11.046996 1506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:11.048766 1506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:11.050098 1506 hybrid_clock.cc:648] HybridClock initialized: now 1781895491050038 us; error 79 us; skew 500 ppm
I20260619 18:58:11.050606 1506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:11 dist-test-slave-dm3f krb5kdc[1492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895491, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:11.054404 1506 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:58:11.056396 1506 webserver.cc:460] Webserver started at http://127.15.91.62:36457/ using document root <none> and password file <none>
I20260619 18:58:11.057354 1506 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:11.057555 1506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:11.057916 1506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:11.061520 1506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "d0060e68130743d997a18ed88e7f68b6"
format_stamp: "Formatted at 2026-06-19 18:58:11 on dist-test-slave-dm3f"
server_key: "a0d85b7fbeb5f0e6d47c3544b604eadc"
server_key_iv: "b0bf966ed9404298dcb449b316f56895"
server_key_version: "encryptionkey@0"
I20260619 18:58:11.062536 1506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "d0060e68130743d997a18ed88e7f68b6"
format_stamp: "Formatted at 2026-06-19 18:58:11 on dist-test-slave-dm3f"
server_key: "a0d85b7fbeb5f0e6d47c3544b604eadc"
server_key_iv: "b0bf966ed9404298dcb449b316f56895"
server_key_version: "encryptionkey@0"
I20260619 18:58:11.069061 1506 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 18:58:11.073628 1521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:11.075556 1506 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:58:11.075848 1506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "d0060e68130743d997a18ed88e7f68b6"
format_stamp: "Formatted at 2026-06-19 18:58:11 on dist-test-slave-dm3f"
server_key: "a0d85b7fbeb5f0e6d47c3544b604eadc"
server_key_iv: "b0bf966ed9404298dcb449b316f56895"
server_key_version: "encryptionkey@0"
I20260619 18:58:11.076133 1506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:11.102552 1506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:11.107702 1506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:11.108052 1506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:11.129437 1506 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:46257
I20260619 18:58:11.129460 1572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:46257 every 8 connection(s)
I20260619 18:58:11.131254 1506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:11.133936 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 1506
I20260619 18:58:11.134181 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:11.134658 15724 external_mini_cluster.cc:1428] Setting key 8af27155949fdaccfe561f6e9c2ec0f6
I20260619 18:58:11.136366 1573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:58:11 dist-test-slave-dm3f krb5kdc[1492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:11.152783 1573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6: Bootstrap starting.
I20260619 18:58:11.157145 1573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:11.158782 1573 log.cc:824] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:11.162855 1573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6: No bootstrap required, opened a new log
I20260619 18:58:11.163256 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:11.136824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.15.91.62:46257)
0619 18:58:11.137851 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:11.137869 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:11.137931 (+ 62us) server_negotiation.cc:407] Connection header received
0619 18:58:11.140192 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:11.140229 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:11.140912 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:11.141456 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:11.143026 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:11.145277 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:11.147003 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:11.148215 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:11.148477 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:11.153607 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:11.153618 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:11.153636 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:11.153688 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:11.157424 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:11.158532 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:11.158539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:11.158547 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:11.158863 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:11.159404 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:11.159410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:11.159421 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:11.160458 (+ 1037us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:11.160815 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:11.161403 (+ 588us) server_negotiation.cc:299] Negotiation successful
0619 18:58:11.161917 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":393,"threads_started":1}
I20260619 18:58:11.172210 1573 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0060e68130743d997a18ed88e7f68b6" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46257 } }
I20260619 18:58:11.172833 1573 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:11.172976 1573 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0060e68130743d997a18ed88e7f68b6, State: Initialized, Role: FOLLOWER
I20260619 18:58:11.173684 1573 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0060e68130743d997a18ed88e7f68b6" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46257 } }
I20260619 18:58:11.173964 1573 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:11.174129 1573 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:11.174330 1573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:11.177224 1573 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0060e68130743d997a18ed88e7f68b6" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46257 } }
I20260619 18:58:11.177968 1573 leader_election.cc:302] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0060e68130743d997a18ed88e7f68b6; no voters:
I20260619 18:58:11.178658 1573 leader_election.cc:288] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:11.178893 1578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:11.179731 1578 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [term 1 LEADER]: Becoming Leader. State: Replica: d0060e68130743d997a18ed88e7f68b6, State: Running, Role: LEADER
I20260619 18:58:11.180397 1578 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0060e68130743d997a18ed88e7f68b6" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46257 } }
I20260619 18:58:11.181354 1573 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:11.184080 1579 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0060e68130743d997a18ed88e7f68b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0060e68130743d997a18ed88e7f68b6" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46257 } } }
I20260619 18:58:11.184489 1579 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:11.183899 1580 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0060e68130743d997a18ed88e7f68b6. Latest consensus state: current_term: 1 leader_uuid: "d0060e68130743d997a18ed88e7f68b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0060e68130743d997a18ed88e7f68b6" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46257 } } }
I20260619 18:58:11.184729 1580 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:11.188082 1586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:11.193857 1586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:11.205647 1586 catalog_manager.cc:1347] Generated new cluster ID: fb831cd0aa8a454ab5ee10b7cd230f9b
I20260619 18:58:11.205807 1586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:11.223847 1586 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:11.225654 1586 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:11.239991 1586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6: Generated new TSK 0
I20260619 18:58:11.241108 1586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:58:11.313874 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46257
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:11.718636 1599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:11.719090 1599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:11.719235 1599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:11.727398 1599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:11.727552 1599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:11.727746 1599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:58:11.738767 1599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:11.740310 1599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:11.741853 1599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:11.757731 1607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:11.757761 1604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:11.757906 1605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:11.759410 1599 server_base.cc:1034] running on GCE node
I20260619 18:58:11.760242 1599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:11.761672 1599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:11.762974 1599 hybrid_clock.cc:648] HybridClock initialized: now 1781895491762914 us; error 75 us; skew 500 ppm
I20260619 18:58:11.763398 1599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:11 dist-test-slave-dm3f krb5kdc[1492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895491, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:11.767349 1599 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:58:11.769361 1599 webserver.cc:460] Webserver started at http://127.15.91.1:42089/ using document root <none> and password file <none>
I20260619 18:58:11.770218 1599 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:11.770359 1599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:11.770659 1599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:11.774261 1599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "205ef8544e304429a808c2494b119d91"
format_stamp: "Formatted at 2026-06-19 18:58:11 on dist-test-slave-dm3f"
server_key: "93cf4ad0c24b09b35e8d63b41edac163"
server_key_iv: "53a70e5e86fb134ebebaeb43a68ccfd3"
server_key_version: "encryptionkey@0"
I20260619 18:58:11.775486 1599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "205ef8544e304429a808c2494b119d91"
format_stamp: "Formatted at 2026-06-19 18:58:11 on dist-test-slave-dm3f"
server_key: "93cf4ad0c24b09b35e8d63b41edac163"
server_key_iv: "53a70e5e86fb134ebebaeb43a68ccfd3"
server_key_version: "encryptionkey@0"
I20260619 18:58:11.782814 1599 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260619 18:58:11.787793 1614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:11.789695 1599 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260619 18:58:11.789927 1599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "205ef8544e304429a808c2494b119d91"
format_stamp: "Formatted at 2026-06-19 18:58:11 on dist-test-slave-dm3f"
server_key: "93cf4ad0c24b09b35e8d63b41edac163"
server_key_iv: "53a70e5e86fb134ebebaeb43a68ccfd3"
server_key_version: "encryptionkey@0"
I20260619 18:58:11.790161 1599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:11.803396 1599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:11.808645 1599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:11.808934 1599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:11.809993 1599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:11.811679 1599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:11.811774 1599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:11.811874 1599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:11.811923 1599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:11.849532 1599 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:46803
I20260619 18:58:11.849582 1726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:46803 every 8 connection(s)
I20260619 18:58:11.851119 1599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:58:11.855672 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 1599
I20260619 18:58:11.855875 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:58:11.856467 15724 external_mini_cluster.cc:1428] Setting key b9e560fae861239974a7499e34f0eb49
Jun 19 18:58:11 dist-test-slave-dm3f krb5kdc[1492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895491, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:11.875443 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:11.854129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:45599 (local address 127.15.91.62:46257)
0619 18:58:11.854684 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:11.854688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:11.856128 (+ 1440us) server_negotiation.cc:407] Connection header received
0619 18:58:11.858775 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:11.858781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:11.858873 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:11.859011 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:11.860889 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:11.861651 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:11.863630 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:11.864062 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:11.864234 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:11.868804 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:11.868811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:11.868814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:11.868857 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:11.871772 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:11.872755 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:11.872760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:11.872763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:11.872903 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:11.873420 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:11.873431 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:11.873434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:11.873733 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:11.873905 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:11.874766 (+ 861us) server_negotiation.cc:299] Negotiation successful
0619 18:58:11.875044 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":281,"threads_started":1}
I20260619 18:58:11.876564 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:11.854846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46257 (local address 127.15.91.1:45599)
0619 18:58:11.855857 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:11.855926 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:11.858367 (+ 2441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:11.859326 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:11.859337 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:11.859951 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:11.860641 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:11.860663 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:11.861884 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:11.861889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:11.863408 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:11.863433 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:11.864363 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:11.864371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:11.864657 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:11.865503 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:58:11.865529 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:11.868462 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:11.872003 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:11.872016 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:11.872044 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:11.872577 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:11.873039 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:11.873044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:11.873048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:11.873283 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:11.873881 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:11.873891 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:11.874302 (+ 411us) client_negotiation.cc:769] Sending connection context
0619 18:58:11.874597 (+ 295us) client_negotiation.cc:240] Negotiation successful
0619 18:58:11.875261 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":388,"threads_started":1}
I20260619 18:58:11.878552 1727 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46257
I20260619 18:58:11.878952 1727 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:11.879797 1727 heartbeater.cc:511] Master 127.15.91.62:46257 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:58:11.882613 1538 ts_manager.cc:194] Registered new tserver with Master: 205ef8544e304429a808c2494b119d91 (127.15.91.1:46803)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
I20260619 18:58:11.884757 1538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:45599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:58:11.916850 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46257
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:12.286165 1733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:12.286535 1733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:12.286764 1733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:12.294844 1733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:12.295028 1733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:12.295229 1733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:58:12.305807 1733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:12.307436 1733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:12.308954 1733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:12.324349 1738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:12.325326 1741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:12.325377 1733 server_base.cc:1034] running on GCE node
W20260619 18:58:12.325466 1739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:12.326277 1733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:12.327603 1733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:12.328927 1733 hybrid_clock.cc:648] HybridClock initialized: now 1781895492328858 us; error 63 us; skew 500 ppm
I20260619 18:58:12.329360 1733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:12 dist-test-slave-dm3f krb5kdc[1492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895492, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:12.333379 1733 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:58:12.335572 1733 webserver.cc:460] Webserver started at http://127.15.91.2:39799/ using document root <none> and password file <none>
I20260619 18:58:12.336438 1733 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:12.336627 1733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:12.337018 1733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:12.340545 1733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "b2583c6020f6496da8f608bafc0888d8"
format_stamp: "Formatted at 2026-06-19 18:58:12 on dist-test-slave-dm3f"
server_key: "274fdfb4f4b8ef5d74892d1d1a1512b1"
server_key_iv: "66c29785991688087bc7a06d911f9f1f"
server_key_version: "encryptionkey@0"
I20260619 18:58:12.341517 1733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "b2583c6020f6496da8f608bafc0888d8"
format_stamp: "Formatted at 2026-06-19 18:58:12 on dist-test-slave-dm3f"
server_key: "274fdfb4f4b8ef5d74892d1d1a1512b1"
server_key_iv: "66c29785991688087bc7a06d911f9f1f"
server_key_version: "encryptionkey@0"
I20260619 18:58:12.348088 1733 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:58:12.352661 1748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:12.354439 1733 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:58:12.354704 1733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "b2583c6020f6496da8f608bafc0888d8"
format_stamp: "Formatted at 2026-06-19 18:58:12 on dist-test-slave-dm3f"
server_key: "274fdfb4f4b8ef5d74892d1d1a1512b1"
server_key_iv: "66c29785991688087bc7a06d911f9f1f"
server_key_version: "encryptionkey@0"
I20260619 18:58:12.354998 1733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:12.376778 1733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:12.382934 1733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:12.383402 1733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:12.385689 1733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:12.388839 1733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:12.388988 1733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:12.389154 1733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:12.389225 1733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:12.434650 1733 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:38873
I20260619 18:58:12.434703 1860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:38873 every 8 connection(s)
I20260619 18:58:12.436360 1733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:58:12.438771 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 1733
I20260619 18:58:12.439004 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:58:12.439560 15724 external_mini_cluster.cc:1428] Setting key 0d65f59ede92c5775ea30737303f389b
Jun 19 18:58:12 dist-test-slave-dm3f krb5kdc[1492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895492, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:12.460952 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:12.440068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:60129 (local address 127.15.91.62:46257)
0619 18:58:12.440638 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:12.440643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:12.442207 (+ 1564us) server_negotiation.cc:407] Connection header received
0619 18:58:12.444460 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:12.444466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:12.444582 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:12.444712 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:12.446551 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:12.447349 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:12.449154 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:12.449623 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:12.449785 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:12.453879 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:12.453884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:12.453888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:12.453928 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:12.457007 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:12.457990 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:12.457998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:12.458001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:12.458157 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:12.458792 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:12.458797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:12.458799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:12.459120 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:12.459334 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:12.460248 (+ 914us) server_negotiation.cc:299] Negotiation successful
0619 18:58:12.460536 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":288,"threads_started":1}
I20260619 18:58:12.462082 1864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:12.440819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46257 (local address 127.15.91.2:60129)
0619 18:58:12.442012 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:12.442078 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:12.444082 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:12.445010 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:12.445020 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:12.445635 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:12.446293 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:12.446314 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:12.447555 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:12.447560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:12.448937 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:12.448953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:12.449916 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:12.449924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:12.450203 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:12.451077 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:58:12.451109 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:12.453546 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:12.457257 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:12.457275 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:12.457291 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:12.457826 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:12.458300 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:12.458306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:12.458310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:12.458576 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:12.459277 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:12.459287 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:12.459783 (+ 496us) client_negotiation.cc:769] Sending connection context
0619 18:58:12.460132 (+ 349us) client_negotiation.cc:240] Negotiation successful
0619 18:58:12.460772 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":492,"threads_started":1}
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
I20260619 18:58:12.463945 1861 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46257
I20260619 18:58:12.464318 1861 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:12.465215 1861 heartbeater.cc:511] Master 127.15.91.62:46257 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
I20260619 18:58:12.467232 1538 ts_manager.cc:194] Registered new tserver with Master: b2583c6020f6496da8f608bafc0888d8 (127.15.91.2:38873)
I20260619 18:58:12.468623 1538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:60129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:58:12.498288 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46257
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:12.865145 1867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:12.865479 1867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:12.865581 1867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:12.875592 1867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:12.875860 1867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:12.876173 1867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:58:12.888558 1867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:46257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:12.889986 1867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:12.889930 1727 heartbeater.cc:503] Master 127.15.91.62:46257 was elected leader, sending a full tablet report...
I20260619 18:58:12.891939 1867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:12.909296 1875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:12.909281 1872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:12.909278 1873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:12.910382 1867 server_base.cc:1034] running on GCE node
I20260619 18:58:12.911280 1867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:12.912431 1867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:12.913757 1867 hybrid_clock.cc:648] HybridClock initialized: now 1781895492913709 us; error 66 us; skew 500 ppm
I20260619 18:58:12.914227 1867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:12 dist-test-slave-dm3f krb5kdc[1492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895492, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:12.919142 1867 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:58:12.921115 1867 webserver.cc:460] Webserver started at http://127.15.91.3:46189/ using document root <none> and password file <none>
I20260619 18:58:12.921983 1867 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:12.922171 1867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:12.922550 1867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:12.926003 1867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "d4150a3d989c43d3a30fcc7584879c7f"
format_stamp: "Formatted at 2026-06-19 18:58:12 on dist-test-slave-dm3f"
server_key: "ac6a3138425ab46b041f739faf1fd3c2"
server_key_iv: "04e2f0d34481829f9853150c2c56dbeb"
server_key_version: "encryptionkey@0"
I20260619 18:58:12.927107 1867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "d4150a3d989c43d3a30fcc7584879c7f"
format_stamp: "Formatted at 2026-06-19 18:58:12 on dist-test-slave-dm3f"
server_key: "ac6a3138425ab46b041f739faf1fd3c2"
server_key_iv: "04e2f0d34481829f9853150c2c56dbeb"
server_key_version: "encryptionkey@0"
I20260619 18:58:12.934031 1867 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:58:12.939255 1882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:12.941596 1867 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:58:12.941859 1867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "d4150a3d989c43d3a30fcc7584879c7f"
format_stamp: "Formatted at 2026-06-19 18:58:12 on dist-test-slave-dm3f"
server_key: "ac6a3138425ab46b041f739faf1fd3c2"
server_key_iv: "04e2f0d34481829f9853150c2c56dbeb"
server_key_version: "encryptionkey@0"
I20260619 18:58:12.942124 1867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:12.975020 1867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:12.980461 1867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:12.980818 1867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:12.982056 1867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:12.983860 1867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:12.983989 1867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:12.984148 1867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:12.984232 1867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:13.024610 1867 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:43657
I20260619 18:58:13.024623 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:43657 every 8 connection(s)
I20260619 18:58:13.026238 1867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:58:13.027431 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 1867
I20260619 18:58:13.027673 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:58:13.028246 15724 external_mini_cluster.cc:1428] Setting key 86401b1268709e412e3559b58535f9e8
Jun 19 18:58:13 dist-test-slave-dm3f krb5kdc[1492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895492, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:13.052347 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.029840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:49913 (local address 127.15.91.62:46257)
0619 18:58:13.030388 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:13.030394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:13.031928 (+ 1534us) server_negotiation.cc:407] Connection header received
0619 18:58:13.034022 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:13.034028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:13.034154 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:13.034287 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:13.036602 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.037493 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.039555 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.040005 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.040173 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.044539 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:13.044545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:13.044549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:13.044600 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:13.047980 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:13.049028 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:13.049033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:13.049036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:13.049243 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:13.049788 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:13.049793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:13.049796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:13.050128 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:13.050284 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:13.051729 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0619 18:58:13.051966 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":281,"threads_started":1}
I20260619 18:58:13.053362 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.030813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46257 (local address 127.15.91.3:49913)
0619 18:58:13.031751 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:13.031804 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:13.033658 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:13.034582 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:13.034595 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:13.035285 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:13.036184 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.036210 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.037717 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.037722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.039317 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.039339 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.040322 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.040328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.040585 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.041370 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:58:13.041396 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:13.044269 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:13.048220 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:13.048238 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:13.048256 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:13.048824 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:13.049371 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:13.049375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:13.049377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:13.049612 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:13.050431 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:13.050442 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:13.051228 (+ 786us) client_negotiation.cc:769] Sending connection context
0619 18:58:13.051527 (+ 299us) client_negotiation.cc:240] Negotiation successful
0619 18:58:13.052013 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":372,"threads_started":1}
I20260619 18:58:13.055339 1995 heartbeater.cc:347] Connected to a master server at 127.15.91.62:46257
I20260619 18:58:13.055764 1995 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:13.056725 1995 heartbeater.cc:511] Master 127.15.91.62:46257 requested a full tablet report, sending...
I20260619 18:58:13.058558 1538 ts_manager.cc:194] Registered new tserver with Master: d4150a3d989c43d3a30fcc7584879c7f (127.15.91.3:43657)
I20260619 18:58:13.059595 1538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:49913
I20260619 18:58:13.070899 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:58:13.091759 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.075837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.15.91.62:46257)
0619 18:58:13.076063 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:13.076071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:13.076511 (+ 440us) server_negotiation.cc:407] Connection header received
0619 18:58:13.076741 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:13.076747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:13.076823 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:13.076936 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:13.078536 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.079386 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.080990 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.081466 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.081629 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.084906 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:13.084912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:13.084915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:13.084961 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:13.088191 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:13.089311 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:13.089315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:13.089318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:13.089479 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:13.089975 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:13.089978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:13.089980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:13.090355 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:13.090606 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:13.091143 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 18:58:13.091324 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:58:13.096921 1538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:58:13.104525 1538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:58:13.131560 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.120220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.15.91.2:38873)
0619 18:58:13.121067 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:13.121075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:13.121110 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:58:13.121582 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:13.121589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:13.121932 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:13.122197 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:13.123272 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.124933 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.128014 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.129705 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.129830 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.130080 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:13.130960 (+ 880us) server_negotiation.cc:299] Negotiation successful
0619 18:58:13.131184 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":546,"threads_started":1}
I20260619 18:58:13.132223 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.121439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:46803 (local address 127.0.0.1:50988)
0619 18:58:13.122102 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:13.122124 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:13.122247 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:13.123938 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:13.123943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:13.123971 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:13.124425 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.124439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.126485 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.126491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.128674 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.128693 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.131196 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.131202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.131417 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.131451 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:58:13.131558 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 18:58:13.131808 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":331,"threads_started":1}
I20260619 18:58:13.132946 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.121805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.15.91.1:46803)
0619 18:58:13.122963 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:13.122973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:13.123021 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:58:13.123141 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:13.123153 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:13.123447 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:13.123675 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:13.124633 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.126276 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.128924 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.130578 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.130751 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.131018 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:13.132369 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0619 18:58:13.132588 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":1126,"threads_started":1}
I20260619 18:58:13.132966 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.120079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38873 (local address 127.0.0.1:57868)
0619 18:58:13.120887 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:13.120936 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:13.121142 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:13.122444 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:13.122453 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:13.122494 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:13.123020 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.123039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.125142 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.125147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.127762 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.127785 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.130323 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.130331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.130584 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.130689 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 18:58:13.131904 (+ 1215us) client_negotiation.cc:240] Negotiation successful
0619 18:58:13.132057 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":379,"threads_started":1}
I20260619 18:58:13.139238 1796 tablet_service.cc:1467] Processing CreateTablet for tablet 263b7e06a3b545a38e2ea239e62026ec (DEFAULT_TABLE table=test-table [id=c51a12d954be44d0a317f7e71da8c1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:13.141070 1796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263b7e06a3b545a38e2ea239e62026ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:13.140862 1662 tablet_service.cc:1467] Processing CreateTablet for tablet 263b7e06a3b545a38e2ea239e62026ec (DEFAULT_TABLE table=test-table [id=c51a12d954be44d0a317f7e71da8c1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:13.143038 1662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263b7e06a3b545a38e2ea239e62026ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:13.144084 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.122485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:43657 (local address 127.0.0.1:48914)
0619 18:58:13.132501 (+ 10016us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:13.132526 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:13.132654 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:13.135156 (+ 2502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:13.135167 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:13.135199 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:13.135442 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.135462 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.138235 (+ 2773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.138244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.140743 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.140777 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.143187 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.143196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.143422 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.143471 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 18:58:13.143574 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:58:13.143755 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9885,"thread_start_us":10874,"threads_started":1}
I20260619 18:58:13.145419 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.122597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.15.91.3:43657)
0619 18:58:13.134392 (+ 11795us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:13.134402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:13.134446 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:58:13.134568 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:13.134577 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:13.134902 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:13.135155 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:13.135671 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.137604 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.141153 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.142867 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.144409 (+ 1542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.144691 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:13.144796 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 18:58:13.144977 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11596,"thread_start_us":11434,"threads_started":1}
I20260619 18:58:13.156076 1930 tablet_service.cc:1467] Processing CreateTablet for tablet 263b7e06a3b545a38e2ea239e62026ec (DEFAULT_TABLE table=test-table [id=c51a12d954be44d0a317f7e71da8c1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:13.158123 1930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263b7e06a3b545a38e2ea239e62026ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:13.158700 2014 tablet_bootstrap.cc:492] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: Bootstrap starting.
I20260619 18:58:13.160331 2015 tablet_bootstrap.cc:492] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91: Bootstrap starting.
I20260619 18:58:13.164695 2014 tablet_bootstrap.cc:654] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:13.164701 2015 tablet_bootstrap.cc:654] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:13.166278 2014 log.cc:824] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:13.166278 2015 log.cc:824] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:13.171252 2015 tablet_bootstrap.cc:492] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91: No bootstrap required, opened a new log
I20260619 18:58:13.171495 2014 tablet_bootstrap.cc:492] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: No bootstrap required, opened a new log
I20260619 18:58:13.171633 2015 ts_tablet_manager.cc:1397] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260619 18:58:13.171802 2014 ts_tablet_manager.cc:1397] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260619 18:58:13.174933 2018 tablet_bootstrap.cc:492] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: Bootstrap starting.
I20260619 18:58:13.179466 2018 tablet_bootstrap.cc:654] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:13.181106 2018 log.cc:824] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:13.182411 2014 raft_consensus.cc:348] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2583c6020f6496da8f608bafc0888d8" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38873 } } peers { permanent_uuid: "205ef8544e304429a808c2494b119d91" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 46803 } } peers { permanent_uuid: "d4150a3d989c43d3a30fcc7584879c7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43657 } }
I20260619 18:58:13.183073 2014 raft_consensus.cc:374] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:13.183224 2014 raft_consensus.cc:729] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2583c6020f6496da8f608bafc0888d8, State: Initialized, Role: FOLLOWER
I20260619 18:58:13.183995 2014 consensus_queue.cc:260] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2583c6020f6496da8f608bafc0888d8" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38873 } } peers { permanent_uuid: "205ef8544e304429a808c2494b119d91" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 46803 } } peers { permanent_uuid: "d4150a3d989c43d3a30fcc7584879c7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43657 } }
I20260619 18:58:13.186085 2018 tablet_bootstrap.cc:492] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: No bootstrap required, opened a new log
I20260619 18:58:13.185629 2015 raft_consensus.cc:348] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2583c6020f6496da8f608bafc0888d8" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38873 } } peers { permanent_uuid: "205ef8544e304429a808c2494b119d91" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 46803 } } peers { permanent_uuid: "d4150a3d989c43d3a30fcc7584879c7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43657 } }
I20260619 18:58:13.186278 2015 raft_consensus.cc:374] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:13.186393 2015 raft_consensus.cc:729] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205ef8544e304429a808c2494b119d91, State: Initialized, Role: FOLLOWER
I20260619 18:58:13.186509 2018 ts_tablet_manager.cc:1397] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260619 18:58:13.187484 1861 heartbeater.cc:503] Master 127.15.91.62:46257 was elected leader, sending a full tablet report...
I20260619 18:58:13.187484 2015 consensus_queue.cc:260] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2583c6020f6496da8f608bafc0888d8" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38873 } } peers { permanent_uuid: "205ef8544e304429a808c2494b119d91" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 46803 } } peers { permanent_uuid: "d4150a3d989c43d3a30fcc7584879c7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43657 } }
I20260619 18:58:13.188253 2014 ts_tablet_manager.cc:1428] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260619 18:58:13.189534 1862 tablet.cc:2369] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:58:13.192374 2015 ts_tablet_manager.cc:1428] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91: Time spent starting tablet: real 0.021s user 0.011s sys 0.005s
I20260619 18:58:13.199143 2018 raft_consensus.cc:348] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2583c6020f6496da8f608bafc0888d8" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38873 } } peers { permanent_uuid: "205ef8544e304429a808c2494b119d91" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 46803 } } peers { permanent_uuid: "d4150a3d989c43d3a30fcc7584879c7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43657 } }
I20260619 18:58:13.199656 2018 raft_consensus.cc:374] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:13.199788 2018 raft_consensus.cc:729] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4150a3d989c43d3a30fcc7584879c7f, State: Initialized, Role: FOLLOWER
I20260619 18:58:13.200474 2018 consensus_queue.cc:260] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2583c6020f6496da8f608bafc0888d8" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38873 } } peers { permanent_uuid: "205ef8544e304429a808c2494b119d91" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 46803 } } peers { permanent_uuid: "d4150a3d989c43d3a30fcc7584879c7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43657 } }
I20260619 18:58:13.201761 1995 heartbeater.cc:503] Master 127.15.91.62:46257 was elected leader, sending a full tablet report...
I20260619 18:58:13.202378 2018 ts_tablet_manager.cc:1428] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260619 18:58:13.216254 2021 raft_consensus.cc:482] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:58:13.216795 2021 raft_consensus.cc:504] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2583c6020f6496da8f608bafc0888d8" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38873 } } peers { permanent_uuid: "205ef8544e304429a808c2494b119d91" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 46803 } } peers { permanent_uuid: "d4150a3d989c43d3a30fcc7584879c7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43657 } }
I20260619 18:58:13.219684 2021 leader_election.cc:288] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2583c6020f6496da8f608bafc0888d8 (127.15.91.2:38873), d4150a3d989c43d3a30fcc7584879c7f (127.15.91.3:43657)
I20260619 18:58:13.228463 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.220055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:38873 (local address 127.15.91.1:45429)
0619 18:58:13.220820 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:13.220839 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:13.221037 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:13.221711 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:13.221721 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:13.221749 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:13.222160 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.222176 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.223869 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.223875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.225730 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.225750 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.227507 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.227515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.227730 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.227782 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 18:58:13.227924 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 18:58:13.228110 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":407,"threads_started":1}
I20260619 18:58:13.228545 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.220524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:45429 (local address 127.15.91.2:38873)
0619 18:58:13.220809 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:13.220816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:13.220943 (+ 127us) server_negotiation.cc:407] Connection header received
0619 18:58:13.221284 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:13.221291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:13.221396 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:13.221511 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:13.222610 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.223675 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.226004 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.227202 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.227340 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.227428 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:13.228035 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 18:58:13.228243 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:58:13.229041 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.221418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:55965 (local address 127.15.91.3:43657)
0619 18:58:13.221653 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:13.221659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:13.221933 (+ 274us) server_negotiation.cc:407] Connection header received
0619 18:58:13.222092 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:13.222097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:13.222184 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:13.222298 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:13.223381 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.224455 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.226662 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.227884 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.228021 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.228127 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:13.228601 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 18:58:13.228784 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 18:58:13.229144 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.221076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:43657 (local address 127.15.91.1:55965)
0619 18:58:13.221816 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:13.221840 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:13.221954 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:13.222620 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:13.222626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:13.222656 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:13.223146 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.223159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.224666 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.224670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.226406 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:13.226426 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.228162 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.228169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:13.228360 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.228407 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 18:58:13.228534 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 18:58:13.228769 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":324,"threads_started":1}
I20260619 18:58:13.230166 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b7e06a3b545a38e2ea239e62026ec" candidate_uuid: "205ef8544e304429a808c2494b119d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2583c6020f6496da8f608bafc0888d8" is_pre_election: true
I20260619 18:58:13.230165 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b7e06a3b545a38e2ea239e62026ec" candidate_uuid: "205ef8544e304429a808c2494b119d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4150a3d989c43d3a30fcc7584879c7f" is_pre_election: true
I20260619 18:58:13.230814 1950 raft_consensus.cc:2454] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 205ef8544e304429a808c2494b119d91 in term 0.
I20260619 18:58:13.230832 1816 raft_consensus.cc:2454] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 205ef8544e304429a808c2494b119d91 in term 0.
I20260619 18:58:13.231645 1615 leader_election.cc:302] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205ef8544e304429a808c2494b119d91, d4150a3d989c43d3a30fcc7584879c7f; no voters:
I20260619 18:58:13.232112 2021 raft_consensus.cc:2781] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:58:13.232280 2021 raft_consensus.cc:482] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:58:13.232391 2021 raft_consensus.cc:3037] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:13.235160 2021 raft_consensus.cc:504] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2583c6020f6496da8f608bafc0888d8" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38873 } } peers { permanent_uuid: "205ef8544e304429a808c2494b119d91" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 46803 } } peers { permanent_uuid: "d4150a3d989c43d3a30fcc7584879c7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43657 } }
I20260619 18:58:13.235983 2021 leader_election.cc:288] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [CANDIDATE]: Term 1 election: Requested vote from peers b2583c6020f6496da8f608bafc0888d8 (127.15.91.2:38873), d4150a3d989c43d3a30fcc7584879c7f (127.15.91.3:43657)
I20260619 18:58:13.236593 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b7e06a3b545a38e2ea239e62026ec" candidate_uuid: "205ef8544e304429a808c2494b119d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2583c6020f6496da8f608bafc0888d8"
I20260619 18:58:13.236652 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b7e06a3b545a38e2ea239e62026ec" candidate_uuid: "205ef8544e304429a808c2494b119d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4150a3d989c43d3a30fcc7584879c7f"
I20260619 18:58:13.236857 1816 raft_consensus.cc:3037] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:13.236892 1950 raft_consensus.cc:3037] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:13.239630 1816 raft_consensus.cc:2454] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 205ef8544e304429a808c2494b119d91 in term 1.
I20260619 18:58:13.239629 1950 raft_consensus.cc:2454] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 205ef8544e304429a808c2494b119d91 in term 1.
I20260619 18:58:13.240283 1615 leader_election.cc:302] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205ef8544e304429a808c2494b119d91, b2583c6020f6496da8f608bafc0888d8; no voters:
I20260619 18:58:13.240688 2021 raft_consensus.cc:2781] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:13.241263 2021 raft_consensus.cc:686] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 1 LEADER]: Becoming Leader. State: Replica: 205ef8544e304429a808c2494b119d91, State: Running, Role: LEADER
I20260619 18:58:13.241915 2021 consensus_queue.cc:237] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2583c6020f6496da8f608bafc0888d8" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38873 } } peers { permanent_uuid: "205ef8544e304429a808c2494b119d91" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 46803 } } peers { permanent_uuid: "d4150a3d989c43d3a30fcc7584879c7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43657 } }
I20260619 18:58:13.247308 1537 catalog_manager.cc:5495] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 205ef8544e304429a808c2494b119d91 (127.15.91.1). New cstate: current_term: 1 leader_uuid: "205ef8544e304429a808c2494b119d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2583c6020f6496da8f608bafc0888d8" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "205ef8544e304429a808c2494b119d91" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 46803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4150a3d989c43d3a30fcc7584879c7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43657 } health_report { overall_health: UNKNOWN } } }
I20260619 18:58:13.269915 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.263984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.15.91.1:46803)
0619 18:58:13.264196 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:13.264201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:13.264341 (+ 140us) server_negotiation.cc:407] Connection header received
0619 18:58:13.264528 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:13.264532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:13.264615 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:13.264776 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:58:13.265155 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.265893 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.267555 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.267979 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.268096 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.268748 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:58:13.269300 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:58:13.269456 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:13.269643 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 18:58:13.269689 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260619 18:58:13.278674 1996 tablet.cc:2369] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:58:13.279806 1816 raft_consensus.cc:1261] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8 [term 1 FOLLOWER]: Refusing update from remote peer 205ef8544e304429a808c2494b119d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:13.279836 1950 raft_consensus.cc:1261] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f [term 1 FOLLOWER]: Refusing update from remote peer 205ef8544e304429a808c2494b119d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:13.281075 2025 consensus_queue.cc:1035] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4150a3d989c43d3a30fcc7584879c7f" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 43657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:13.281641 2021 consensus_queue.cc:1035] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2583c6020f6496da8f608bafc0888d8" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:13.302870 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298643940461801472 to 7298643940323794944. Current Snapshot: MvccSnapshot[applied={T|T < 7298643940461801472}]
I20260619 18:58:13.303277 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298643940461801472 to 7298643940323794944. Current Snapshot: MvccSnapshot[applied={T|T < 7298643940461801472 or (T in {7298643940461801472})}]
I20260619 18:58:13.316269 1538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:13.316642 1538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:13.321398 1538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0060e68130743d997a18ed88e7f68b6: Sending DeleteTablet for 3 replicas of tablet 263b7e06a3b545a38e2ea239e62026ec
I20260619 18:58:13.323043 1662 tablet_service.cc:1514] Processing DeleteTablet for tablet 263b7e06a3b545a38e2ea239e62026ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:13 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:50988
I20260619 18:58:13.322989 1796 tablet_service.cc:1514] Processing DeleteTablet for tablet 263b7e06a3b545a38e2ea239e62026ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:13 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:57868
I20260619 18:58:13.323225 1930 tablet_service.cc:1514] Processing DeleteTablet for tablet 263b7e06a3b545a38e2ea239e62026ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:13 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:48914
I20260619 18:58:13.326083 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 1599
I20260619 18:58:13.343011 2037 tablet_replica.cc:330] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: stopping tablet replica
I20260619 18:58:13.343874 2037 raft_consensus.cc:2229] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:58:13.344772 2037 raft_consensus.cc:2258] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:13.346216 2039 tablet_replica.cc:330] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: stopping tablet replica
I20260619 18:58:13.348006 2038 tablet_replica.cc:330] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91: stopping tablet replica
I20260619 18:58:13.348344 2037 ts_tablet_manager.cc:1905] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:13.348543 2038 raft_consensus.cc:2229] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:58:13.349326 2038 raft_consensus.cc:2258] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:13.350035 2039 raft_consensus.cc:2229] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f [term 1 FOLLOWER]: Raft consensus shutting down.
Jun 19 18:58:13 dist-test-slave-dm3f krb5kdc[1492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:58:13.351950 2038 ts_tablet_manager.cc:1905] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:13.352699 2039 raft_consensus.cc:2258] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:13.356160 2039 ts_tablet_manager.cc:1905] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:13.361079 2038 ts_tablet_manager.cc:1918] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:13.361307 2038 log.cc:1196] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/263b7e06a3b545a38e2ea239e62026ec
I20260619 18:58:13.361940 2038 ts_tablet_manager.cc:1939] T 263b7e06a3b545a38e2ea239e62026ec P 205ef8544e304429a808c2494b119d91: Deleting consensus metadata
I20260619 18:58:13.364377 1522 catalog_manager.cc:4841] TS 205ef8544e304429a808c2494b119d91 (127.15.91.1:46803): tablet 263b7e06a3b545a38e2ea239e62026ec (table test-table [id=c51a12d954be44d0a317f7e71da8c1af]) successfully deleted
I20260619 18:58:13.365231 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.328130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.15.91.1:46803)
0619 18:58:13.328392 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:13.328402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:13.331569 (+ 3167us) server_negotiation.cc:407] Connection header received
0619 18:58:13.332610 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:13.332616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:13.332716 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:13.332827 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:13.338259 (+ 5432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.340144 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.345502 (+ 5358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.346405 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.346608 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.353494 (+ 6886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:13.353512 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:13.353538 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:13.353615 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:13.359038 (+ 5423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:13.360461 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:13.360474 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:13.360478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:13.360766 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:13.361384 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:13.361392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:13.361396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:13.361961 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:13.362369 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:13.363615 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0619 18:58:13.363974 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 18:58:13.365775 2037 ts_tablet_manager.cc:1918] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:13.366070 2037 log.cc:1196] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/263b7e06a3b545a38e2ea239e62026ec
I20260619 18:58:13.366791 2037 ts_tablet_manager.cc:1939] T 263b7e06a3b545a38e2ea239e62026ec P b2583c6020f6496da8f608bafc0888d8: Deleting consensus metadata
I20260619 18:58:13.367183 1722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:13.369308 1522 catalog_manager.cc:4841] TS b2583c6020f6496da8f608bafc0888d8 (127.15.91.2:38873): tablet 263b7e06a3b545a38e2ea239e62026ec (table test-table [id=c51a12d954be44d0a317f7e71da8c1af]) successfully deleted
I20260619 18:58:13.371016 2039 ts_tablet_manager.cc:1918] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:13.371236 2039 log.cc:1196] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/263b7e06a3b545a38e2ea239e62026ec
I20260619 18:58:13.371848 2039 ts_tablet_manager.cc:1939] T 263b7e06a3b545a38e2ea239e62026ec P d4150a3d989c43d3a30fcc7584879c7f: Deleting consensus metadata
I20260619 18:58:13.374059 1522 catalog_manager.cc:4841] TS d4150a3d989c43d3a30fcc7584879c7f (127.15.91.3:43657): tablet 263b7e06a3b545a38e2ea239e62026ec (table test-table [id=c51a12d954be44d0a317f7e71da8c1af]) successfully deleted
I20260619 18:58:13.476979 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 1599
I20260619 18:58:13.498309 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 1733
Jun 19 18:58:13 dist-test-slave-dm3f krb5kdc[1492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:58:13.515949 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.499229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.15.91.2:38873)
0619 18:58:13.499437 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:13.499445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:13.499553 (+ 108us) server_negotiation.cc:407] Connection header received
0619 18:58:13.499788 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:13.499794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:13.499878 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:13.499994 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:13.501368 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.502145 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.503947 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.504372 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.504533 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.509456 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:13.509475 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:13.509492 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:13.509544 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:13.512631 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:13.513534 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:13.513544 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:13.513547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:13.513697 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:13.514151 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:13.514155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:13.514157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:13.514578 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:13.514943 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:13.515309 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 18:58:13.515552 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:58:13.517261 1856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:13.611937 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 1733
I20260619 18:58:13.630913 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 1867
Jun 19 18:58:13 dist-test-slave-dm3f krb5kdc[1492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:58:13.649801 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:13.631826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.15.91.3:43657)
0619 18:58:13.632043 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:13.632050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:13.632076 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:58:13.632291 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:13.632296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:13.632367 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:13.632488 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:13.633970 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.634821 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.636373 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:13.636809 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:13.636970 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:13.642975 (+ 6005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:13.642995 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:13.643013 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:13.643059 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:13.646400 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:13.647455 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:13.647464 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:13.647466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:13.647617 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:13.648056 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:13.648062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:13.648065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:13.648477 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:13.648822 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:13.649166 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 18:58:13.649417 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:58:13.650580 1990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:13.747524 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 1867
I20260619 18:58:13.766088 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 1506
I20260619 18:58:13.767635 1568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:13.883569 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 1506
2026-06-19T18:58:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4651 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T18:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:13Z Disabled control of system clock
I20260619 18:58:13.973104 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:39205
--webserver_interface=127.15.91.62
--webserver_port=0
--builtin_ntp_servers=127.15.91.20:42267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:39205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:58:14.365130 2048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:14.365500 2048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:14.365670 2048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:14.375466 2048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:14.375623 2048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:14.375723 2048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:14.375829 2048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:14.375907 2048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:14.386775 2048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:42267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:39205
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:39205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:14.388273 2048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:14.390338 2048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:14.404956 2056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:14.405017 2053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:14.404876 2054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:14.406621 2048 server_base.cc:1034] running on GCE node
I20260619 18:58:14.407961 2048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:14.409814 2048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:14.411154 2048 hybrid_clock.cc:648] HybridClock initialized: now 1781895494411089 us; error 87 us; skew 500 ppm
I20260619 18:58:14.411674 2048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:14.414943 2048 webserver.cc:460] Webserver started at http://127.15.91.62:46055/ using document root <none> and password file <none>
I20260619 18:58:14.415828 2048 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:14.415998 2048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:14.416371 2048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:14.419996 2048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "0e7e6d874bf94310a02ff87632f1edc7"
format_stamp: "Formatted at 2026-06-19 18:58:14 on dist-test-slave-dm3f"
server_key: "cb501ff1d2a4951cb59577705679c57d"
server_key_iv: "0abac37b30be46b7981b772e68e9a9b7"
server_key_version: "encryptionkey@0"
I20260619 18:58:14.421021 2048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "0e7e6d874bf94310a02ff87632f1edc7"
format_stamp: "Formatted at 2026-06-19 18:58:14 on dist-test-slave-dm3f"
server_key: "cb501ff1d2a4951cb59577705679c57d"
server_key_iv: "0abac37b30be46b7981b772e68e9a9b7"
server_key_version: "encryptionkey@0"
I20260619 18:58:14.428146 2048 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260619 18:58:14.432868 2062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:14.434824 2048 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 18:58:14.435089 2048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "0e7e6d874bf94310a02ff87632f1edc7"
format_stamp: "Formatted at 2026-06-19 18:58:14 on dist-test-slave-dm3f"
server_key: "cb501ff1d2a4951cb59577705679c57d"
server_key_iv: "0abac37b30be46b7981b772e68e9a9b7"
server_key_version: "encryptionkey@0"
I20260619 18:58:14.435377 2048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:14.458568 2048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:14.459694 2048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:14.460073 2048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:14.483732 2048 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:39205
I20260619 18:58:14.483799 2113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:39205 every 8 connection(s)
I20260619 18:58:14.485380 2048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:14.489991 2114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:14.494329 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 2048
I20260619 18:58:14.494567 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:14.495131 15724 external_mini_cluster.cc:1428] Setting key e17a35dbf88ebf369fbf5d5a7c53ef57
I20260619 18:58:14.503598 2114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7: Bootstrap starting.
I20260619 18:58:14.508157 2114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:14.509788 2114 log.cc:824] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:14.513973 2114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7: No bootstrap required, opened a new log
I20260619 18:58:14.514086 2117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:14.497465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.15.91.62:39205)
0619 18:58:14.498663 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:14.498677 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:14.498812 (+ 135us) server_negotiation.cc:407] Connection header received
0619 18:58:14.501660 (+ 2848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:14.501699 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:14.502401 (+ 702us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:14.502942 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:14.503616 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:14.505162 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:14.507462 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:14.508828 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:14.509121 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:14.510945 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:14.510959 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:14.510985 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:14.511907 (+ 922us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:14.512011 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:14.512042 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:14.512181 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:14.512499 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 18:58:14.512941 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":461,"threads_started":1}
I20260619 18:58:14.524183 2114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7e6d874bf94310a02ff87632f1edc7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39205 } }
I20260619 18:58:14.524751 2114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:14.524863 2114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7e6d874bf94310a02ff87632f1edc7, State: Initialized, Role: FOLLOWER
I20260619 18:58:14.525624 2114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7e6d874bf94310a02ff87632f1edc7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39205 } }
I20260619 18:58:14.525947 2114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:14.526120 2114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:14.526310 2114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:14.528863 2114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7e6d874bf94310a02ff87632f1edc7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39205 } }
I20260619 18:58:14.529548 2114 leader_election.cc:302] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e7e6d874bf94310a02ff87632f1edc7; no voters:
I20260619 18:58:14.530272 2114 leader_election.cc:288] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:14.530534 2119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:14.531967 2119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [term 1 LEADER]: Becoming Leader. State: Replica: 0e7e6d874bf94310a02ff87632f1edc7, State: Running, Role: LEADER
I20260619 18:58:14.532702 2119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7e6d874bf94310a02ff87632f1edc7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39205 } }
I20260619 18:58:14.532977 2114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:14.536335 2120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e7e6d874bf94310a02ff87632f1edc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7e6d874bf94310a02ff87632f1edc7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39205 } } }
I20260619 18:58:14.536320 2121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e7e6d874bf94310a02ff87632f1edc7. Latest consensus state: current_term: 1 leader_uuid: "0e7e6d874bf94310a02ff87632f1edc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7e6d874bf94310a02ff87632f1edc7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39205 } } }
I20260619 18:58:14.536988 2120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:14.536980 2121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:14.539932 2128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:14.546324 2128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:14.557677 2128 catalog_manager.cc:1347] Generated new cluster ID: 1a43d6caa18b4a4d92aeec7171f74ce4
I20260619 18:58:14.557818 2128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:14.578311 2128 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:14.580111 2128 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:14.588114 2128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7: Generated new TSK 0
I20260619 18:58:14.589442 2128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:14.607460 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39205
--builtin_ntp_servers=127.15.91.20:42267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:58:14.996867 2138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:14.997232 2138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:14.997398 2138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:15.005802 2138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:15.005950 2138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:15.006155 2138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:58:15.016928 2138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:42267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:15.018391 2138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:15.020577 2138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:15.036303 2143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:15.036301 2146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:15.036487 2144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:15.037207 2138 server_base.cc:1034] running on GCE node
I20260619 18:58:15.038126 2138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:15.039506 2138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:15.040841 2138 hybrid_clock.cc:648] HybridClock initialized: now 1781895495040749 us; error 89 us; skew 500 ppm
I20260619 18:58:15.041277 2138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:15.044467 2138 webserver.cc:460] Webserver started at http://127.15.91.1:41299/ using document root <none> and password file <none>
I20260619 18:58:15.045589 2138 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:15.045778 2138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:15.046159 2138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:15.049629 2138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "5e43a09c74c3414d9546f4ad1372b32a"
format_stamp: "Formatted at 2026-06-19 18:58:15 on dist-test-slave-dm3f"
server_key: "55210b86388c3f49a0b2342e8103cb5a"
server_key_iv: "95b534dfe88e0724b11791a796bff812"
server_key_version: "encryptionkey@0"
I20260619 18:58:15.050676 2138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "5e43a09c74c3414d9546f4ad1372b32a"
format_stamp: "Formatted at 2026-06-19 18:58:15 on dist-test-slave-dm3f"
server_key: "55210b86388c3f49a0b2342e8103cb5a"
server_key_iv: "95b534dfe88e0724b11791a796bff812"
server_key_version: "encryptionkey@0"
I20260619 18:58:15.057106 2138 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:58:15.062111 2153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:15.063925 2138 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:15.064219 2138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "5e43a09c74c3414d9546f4ad1372b32a"
format_stamp: "Formatted at 2026-06-19 18:58:15 on dist-test-slave-dm3f"
server_key: "55210b86388c3f49a0b2342e8103cb5a"
server_key_iv: "95b534dfe88e0724b11791a796bff812"
server_key_version: "encryptionkey@0"
I20260619 18:58:15.064524 2138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:15.098702 2138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:15.099759 2138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:15.100095 2138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:15.101281 2138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:15.103156 2138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:15.103327 2138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:15.103470 2138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:15.103564 2138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:15.143963 2138 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:41505
I20260619 18:58:15.144024 2265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:41505 every 8 connection(s)
I20260619 18:58:15.145525 2138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:58:15.147958 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 2138
I20260619 18:58:15.148196 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:58:15.148737 15724 external_mini_cluster.cc:1428] Setting key 7f0b21ac12a615638a981e04ab29e170
I20260619 18:58:15.152925 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39205
--builtin_ntp_servers=127.15.91.20:42267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:58:15.167951 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:15.148892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:39783 (local address 127.15.91.62:39205)
0619 18:58:15.149479 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:15.149485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:15.151488 (+ 2003us) server_negotiation.cc:407] Connection header received
0619 18:58:15.155506 (+ 4018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:15.155511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:15.155607 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:15.155722 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:15.157960 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:15.159201 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:15.162679 (+ 3478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:15.163762 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:15.164188 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:15.166388 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:15.166399 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:15.166408 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:15.166443 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:15.166511 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:15.166534 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:15.166693 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:15.167316 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 18:58:15.167472 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":309,"threads_started":1}
I20260619 18:58:15.169014 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:15.149763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39205 (local address 127.15.91.1:39783)
0619 18:58:15.151259 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:15.151332 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:15.154885 (+ 3553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:15.156127 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:15.156146 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:15.156416 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:15.157524 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:15.157565 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:15.159510 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:15.159520 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:15.162111 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:15.162145 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:15.164259 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:15.164267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:15.164580 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:15.165752 (+ 1172us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:58:15.165756 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:15.165830 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:58:15.165979 (+ 149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:15.166938 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:15.166958 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:15.166969 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:58:15.167230 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 18:58:15.167682 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":410,"threads_started":1}
I20260619 18:58:15.171365 2266 heartbeater.cc:347] Connected to a master server at 127.15.91.62:39205
I20260619 18:58:15.171816 2266 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:15.173143 2266 heartbeater.cc:511] Master 127.15.91.62:39205 requested a full tablet report, sending...
I20260619 18:58:15.176076 2079 ts_manager.cc:194] Registered new tserver with Master: 5e43a09c74c3414d9546f4ad1372b32a (127.15.91.1:41505)
I20260619 18:58:15.177918 2079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.1:39783
W20260619 18:58:15.526414 2270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:15.526818 2270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:15.527009 2270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:15.535100 2270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:15.535233 2270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:15.535351 2270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:58:15.546023 2270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:42267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:15.547567 2270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:15.549552 2270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:15.564543 2278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:15.564507 2276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:15.564577 2275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:15.568549 2270 server_base.cc:1034] running on GCE node
I20260619 18:58:15.569342 2270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:15.570613 2270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:15.571923 2270 hybrid_clock.cc:648] HybridClock initialized: now 1781895495571839 us; error 77 us; skew 500 ppm
I20260619 18:58:15.572340 2270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:15.575356 2270 webserver.cc:460] Webserver started at http://127.15.91.2:44425/ using document root <none> and password file <none>
I20260619 18:58:15.576223 2270 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:15.576341 2270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:15.576748 2270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:15.580339 2270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "6f218cd51c254b87b5f7d760417cfd05"
format_stamp: "Formatted at 2026-06-19 18:58:15 on dist-test-slave-dm3f"
server_key: "6eb66421b010f2748fbbed63398bbe80"
server_key_iv: "ebba099af44a3919214d4ef7213ecb22"
server_key_version: "encryptionkey@0"
I20260619 18:58:15.581336 2270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "6f218cd51c254b87b5f7d760417cfd05"
format_stamp: "Formatted at 2026-06-19 18:58:15 on dist-test-slave-dm3f"
server_key: "6eb66421b010f2748fbbed63398bbe80"
server_key_iv: "ebba099af44a3919214d4ef7213ecb22"
server_key_version: "encryptionkey@0"
I20260619 18:58:15.588354 2270 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260619 18:58:15.592891 2284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:15.594673 2270 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 18:58:15.594983 2270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "6f218cd51c254b87b5f7d760417cfd05"
format_stamp: "Formatted at 2026-06-19 18:58:15 on dist-test-slave-dm3f"
server_key: "6eb66421b010f2748fbbed63398bbe80"
server_key_iv: "ebba099af44a3919214d4ef7213ecb22"
server_key_version: "encryptionkey@0"
I20260619 18:58:15.595299 2270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:15.634465 2270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:15.635520 2270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:15.635865 2270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:15.637198 2270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:15.639066 2270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:15.639232 2270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:15.639371 2270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:15.639469 2270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:15.680426 2270 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:45333
I20260619 18:58:15.680493 2396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:45333 every 8 connection(s)
I20260619 18:58:15.681981 2270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:58:15.688555 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 2270
I20260619 18:58:15.688926 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:58:15.689985 15724 external_mini_cluster.cc:1428] Setting key 449c4e0b9a3ad85ea591c74913a194aa
I20260619 18:58:15.694190 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39205
--builtin_ntp_servers=127.15.91.20:42267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:58:15.700024 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:15.684969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:34821 (local address 127.15.91.62:39205)
0619 18:58:15.685440 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:15.685445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:15.687118 (+ 1673us) server_negotiation.cc:407] Connection header received
0619 18:58:15.690617 (+ 3499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:15.690623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:15.690761 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:15.690922 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:15.692610 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:15.693645 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:15.695917 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:15.696743 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:15.696934 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:15.698766 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:15.698776 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:15.698784 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:15.698818 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:15.698880 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:15.698903 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:15.699020 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:15.699370 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 18:58:15.699514 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
I20260619 18:58:15.701263 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:15.685709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39205 (local address 127.15.91.2:34821)
0619 18:58:15.686900 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:15.686948 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:15.689768 (+ 2820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:15.691256 (+ 1488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:15.691268 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:15.691391 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:15.692356 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:15.692389 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:15.693930 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:15.693936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:15.695656 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:15.695679 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:15.697114 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:15.697122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:15.697394 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:15.698262 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:58:15.698265 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:15.698316 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:58:15.698428 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:15.699065 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:15.699083 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:15.699094 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:58:15.699348 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 18:58:15.699799 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":343,"threads_started":1}
I20260619 18:58:15.703303 2397 heartbeater.cc:347] Connected to a master server at 127.15.91.62:39205
I20260619 18:58:15.703814 2397 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:15.704766 2397 heartbeater.cc:511] Master 127.15.91.62:39205 requested a full tablet report, sending...
I20260619 18:58:15.706928 2079 ts_manager.cc:194] Registered new tserver with Master: 6f218cd51c254b87b5f7d760417cfd05 (127.15.91.2:45333)
I20260619 18:58:15.707864 2079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.2:34821
W20260619 18:58:16.069864 2401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:16.070256 2401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:16.070389 2401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:16.078264 2401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:16.078401 2401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:16.078514 2401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:58:16.088577 2401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:42267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:39205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:16.089931 2401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:16.091946 2401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:16.107203 2406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:16.107271 2409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:16.107175 2407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:16.108341 2401 server_base.cc:1034] running on GCE node
I20260619 18:58:16.109095 2401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:16.110216 2401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:16.111485 2401 hybrid_clock.cc:648] HybridClock initialized: now 1781895496111422 us; error 60 us; skew 500 ppm
I20260619 18:58:16.111882 2401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:16.114943 2401 webserver.cc:460] Webserver started at http://127.15.91.3:38435/ using document root <none> and password file <none>
I20260619 18:58:16.115825 2401 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:16.115999 2401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:16.116400 2401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:16.119813 2401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "06148076a5a04e8d80d886905b616f75"
format_stamp: "Formatted at 2026-06-19 18:58:16 on dist-test-slave-dm3f"
server_key: "73896aa5db02c2d8c2929a92cd3ede1f"
server_key_iv: "99addb02b9dcd3e1f94f8e41416c551b"
server_key_version: "encryptionkey@0"
I20260619 18:58:16.120817 2401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "06148076a5a04e8d80d886905b616f75"
format_stamp: "Formatted at 2026-06-19 18:58:16 on dist-test-slave-dm3f"
server_key: "73896aa5db02c2d8c2929a92cd3ede1f"
server_key_iv: "99addb02b9dcd3e1f94f8e41416c551b"
server_key_version: "encryptionkey@0"
I20260619 18:58:16.127205 2401 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:58:16.131669 2415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:16.133440 2401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:58:16.133690 2401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "06148076a5a04e8d80d886905b616f75"
format_stamp: "Formatted at 2026-06-19 18:58:16 on dist-test-slave-dm3f"
server_key: "73896aa5db02c2d8c2929a92cd3ede1f"
server_key_iv: "99addb02b9dcd3e1f94f8e41416c551b"
server_key_version: "encryptionkey@0"
I20260619 18:58:16.134040 2401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:16.162400 2401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:16.163471 2401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:16.163825 2401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:16.164973 2401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:16.167095 2401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:16.167258 2401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:16.167407 2401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:16.167501 2401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:16.182951 2266 heartbeater.cc:503] Master 127.15.91.62:39205 was elected leader, sending a full tablet report...
I20260619 18:58:16.207953 2401 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:33111
I20260619 18:58:16.208055 2527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:33111 every 8 connection(s)
I20260619 18:58:16.209492 2401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:58:16.217643 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 2401
I20260619 18:58:16.217890 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:58:16.218403 15724 external_mini_cluster.cc:1428] Setting key 59a3408ff128e8f2e8b8b0b8e714f435
I20260619 18:58:16.226409 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.212743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:52483 (local address 127.15.91.62:39205)
0619 18:58:16.213570 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:16.213576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:16.215111 (+ 1535us) server_negotiation.cc:407] Connection header received
0619 18:58:16.217383 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:16.217389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:16.217508 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:16.217638 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:16.219230 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.220507 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.222985 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.223433 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.223596 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.225341 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:16.225347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:16.225351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:16.225377 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:16.225422 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:16.225435 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:16.225545 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:16.225821 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 18:58:16.225949 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":441,"threads_started":1}
I20260619 18:58:16.227670 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.213561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39205 (local address 127.15.91.3:52483)
0619 18:58:16.214883 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:16.214941 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:16.216932 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:16.217998 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:16.218015 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:16.218157 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:16.218979 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.219005 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.220731 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.220737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.222780 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.222812 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.223738 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.223748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.224033 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.224891 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:58:16.224894 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:16.224944 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:58:16.225049 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:16.225550 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:16.225565 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:16.225574 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:58:16.225798 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 18:58:16.226345 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":367,"threads_started":1}
I20260619 18:58:16.229645 2528 heartbeater.cc:347] Connected to a master server at 127.15.91.62:39205
I20260619 18:58:16.230346 2528 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:16.231415 2528 heartbeater.cc:511] Master 127.15.91.62:39205 requested a full tablet report, sending...
I20260619 18:58:16.233855 2078 ts_manager.cc:194] Registered new tserver with Master: 06148076a5a04e8d80d886905b616f75 (127.15.91.3:33111)
I20260619 18:58:16.235240 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:58:16.235239 2078 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.3:52483
I20260619 18:58:16.248237 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.240941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.15.91.62:39205)
0619 18:58:16.241258 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:16.241264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:16.241577 (+ 313us) server_negotiation.cc:407] Connection header received
0619 18:58:16.241699 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:16.241704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:16.241811 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:16.242067 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:16.242699 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.243548 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.245283 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.245742 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.245859 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.247315 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:16.247320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:16.247324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:16.247347 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:16.247384 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:16.247398 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:16.247529 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:16.247810 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 18:58:16.247900 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 18:58:16.253531 2078 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:58:16.261045 2078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:58:16.301641 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.276347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.15.91.3:33111)
0619 18:58:16.290527 (+ 14180us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:16.290538 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:16.290581 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:58:16.290705 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:16.290740 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:16.291041 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:16.291336 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:16.292829 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.295090 (+ 2261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.298340 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.300054 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.300298 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.300600 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:16.300998 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 18:58:16.301232 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13981,"thread_start_us":13799,"threads_started":1}
I20260619 18:58:16.301661 2541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.276225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:33111 (local address 127.0.0.1:48190)
0619 18:58:16.277174 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:16.277229 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:16.277445 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:16.291328 (+ 13883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:16.291332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:16.291355 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:16.292443 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.292462 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.295328 (+ 2866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.295334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.298036 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.298057 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.300391 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.300400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.300672 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.300740 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 18:58:16.300921 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 18:58:16.301162 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":201,"thread_start_us":772,"threads_started":1}
I20260619 18:58:16.302052 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.280482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.15.91.1:41505)
0619 18:58:16.290913 (+ 10431us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:16.290922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:16.290951 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:58:16.291047 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:16.291055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:16.291319 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:16.291710 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:16.292935 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.294373 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.298147 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.300050 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.300239 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.301169 (+ 930us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:16.301244 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 18:58:16.301431 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10212,"thread_start_us":10058,"threads_started":1}
I20260619 18:58:16.302353 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.276309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:45333 (local address 127.0.0.1:59672)
0619 18:58:16.289370 (+ 13061us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:16.289399 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:16.289603 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:16.291017 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:16.291025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:16.291070 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:16.292251 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.292307 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.295398 (+ 3091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.295403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.297499 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.297519 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.301403 (+ 3884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.301410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.301620 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.301669 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 18:58:16.301790 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 18:58:16.301967 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12872,"mutex_wait_us":157,"spinlock_wait_cycles":8960,"thread_start_us":13391,"threads_started":1}
I20260619 18:58:16.302946 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.279773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:41505 (local address 127.0.0.1:55102)
0619 18:58:16.289444 (+ 9671us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:16.289466 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:16.289603 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:16.291575 (+ 1972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:16.291580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:16.291613 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:16.292638 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.292667 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.294559 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.294566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.297423 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.297462 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.300649 (+ 3187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.300665 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.300828 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.300853 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:58:16.300971 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 18:58:16.302512 (+ 1541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9549,"mutex_wait_us":202,"spinlock_wait_cycles":5632,"thread_start_us":9176,"threads_started":1}
I20260619 18:58:16.303216 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.276796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.15.91.2:45333)
0619 18:58:16.288729 (+ 11933us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:16.288745 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:16.289579 (+ 834us) server_negotiation.cc:407] Connection header received
0619 18:58:16.289813 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:16.289827 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:16.290646 (+ 819us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:16.290992 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:16.292802 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.295128 (+ 2326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.297877 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.300861 (+ 2984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.301003 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.301250 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:16.302614 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0619 18:58:16.302849 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11642,"thread_start_us":11669,"threads_started":1}
I20260619 18:58:16.307049 2332 tablet_service.cc:1467] Processing CreateTablet for tablet d0cf7ccecc7043dfb84d73385bea5c57 (DEFAULT_TABLE table=test-table [id=7606101615f84292876d344eb3223bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:16.306887 2463 tablet_service.cc:1467] Processing CreateTablet for tablet d0cf7ccecc7043dfb84d73385bea5c57 (DEFAULT_TABLE table=test-table [id=7606101615f84292876d344eb3223bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:16.306887 2201 tablet_service.cc:1467] Processing CreateTablet for tablet d0cf7ccecc7043dfb84d73385bea5c57 (DEFAULT_TABLE table=test-table [id=7606101615f84292876d344eb3223bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:16.308430 2332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cf7ccecc7043dfb84d73385bea5c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:16.308530 2201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cf7ccecc7043dfb84d73385bea5c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:16.308554 2463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cf7ccecc7043dfb84d73385bea5c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:16.325021 2547 tablet_bootstrap.cc:492] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: Bootstrap starting.
I20260619 18:58:16.328840 2547 tablet_bootstrap.cc:654] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:16.329975 2549 tablet_bootstrap.cc:492] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: Bootstrap starting.
I20260619 18:58:16.330492 2547 log.cc:824] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:16.332481 2548 tablet_bootstrap.cc:492] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: Bootstrap starting.
I20260619 18:58:16.335037 2547 tablet_bootstrap.cc:492] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: No bootstrap required, opened a new log
I20260619 18:58:16.335275 2549 tablet_bootstrap.cc:654] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:16.335429 2547 ts_tablet_manager.cc:1397] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260619 18:58:16.337087 2549 log.cc:824] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:16.337320 2548 tablet_bootstrap.cc:654] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:16.338940 2548 log.cc:824] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:16.341670 2549 tablet_bootstrap.cc:492] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: No bootstrap required, opened a new log
I20260619 18:58:16.342067 2549 ts_tablet_manager.cc:1397] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260619 18:58:16.342962 2548 tablet_bootstrap.cc:492] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: No bootstrap required, opened a new log
I20260619 18:58:16.343317 2548 ts_tablet_manager.cc:1397] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260619 18:58:16.345528 2547 raft_consensus.cc:348] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06148076a5a04e8d80d886905b616f75" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33111 } } peers { permanent_uuid: "6f218cd51c254b87b5f7d760417cfd05" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 45333 } } peers { permanent_uuid: "5e43a09c74c3414d9546f4ad1372b32a" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41505 } }
I20260619 18:58:16.346015 2547 raft_consensus.cc:374] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:16.346118 2547 raft_consensus.cc:729] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f218cd51c254b87b5f7d760417cfd05, State: Initialized, Role: FOLLOWER
I20260619 18:58:16.346947 2547 consensus_queue.cc:260] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06148076a5a04e8d80d886905b616f75" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33111 } } peers { permanent_uuid: "6f218cd51c254b87b5f7d760417cfd05" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 45333 } } peers { permanent_uuid: "5e43a09c74c3414d9546f4ad1372b32a" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41505 } }
I20260619 18:58:16.350816 2397 heartbeater.cc:503] Master 127.15.91.62:39205 was elected leader, sending a full tablet report...
I20260619 18:58:16.351657 2547 ts_tablet_manager.cc:1428] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260619 18:58:16.354494 2549 raft_consensus.cc:348] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06148076a5a04e8d80d886905b616f75" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33111 } } peers { permanent_uuid: "6f218cd51c254b87b5f7d760417cfd05" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 45333 } } peers { permanent_uuid: "5e43a09c74c3414d9546f4ad1372b32a" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41505 } }
I20260619 18:58:16.355129 2549 raft_consensus.cc:374] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:16.355264 2549 raft_consensus.cc:729] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06148076a5a04e8d80d886905b616f75, State: Initialized, Role: FOLLOWER
I20260619 18:58:16.355324 2548 raft_consensus.cc:348] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06148076a5a04e8d80d886905b616f75" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33111 } } peers { permanent_uuid: "6f218cd51c254b87b5f7d760417cfd05" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 45333 } } peers { permanent_uuid: "5e43a09c74c3414d9546f4ad1372b32a" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41505 } }
I20260619 18:58:16.355835 2548 raft_consensus.cc:374] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:16.355948 2548 raft_consensus.cc:729] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e43a09c74c3414d9546f4ad1372b32a, State: Initialized, Role: FOLLOWER
I20260619 18:58:16.356048 2549 consensus_queue.cc:260] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06148076a5a04e8d80d886905b616f75" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33111 } } peers { permanent_uuid: "6f218cd51c254b87b5f7d760417cfd05" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 45333 } } peers { permanent_uuid: "5e43a09c74c3414d9546f4ad1372b32a" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41505 } }
I20260619 18:58:16.356700 2548 consensus_queue.cc:260] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06148076a5a04e8d80d886905b616f75" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33111 } } peers { permanent_uuid: "6f218cd51c254b87b5f7d760417cfd05" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 45333 } } peers { permanent_uuid: "5e43a09c74c3414d9546f4ad1372b32a" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41505 } }
I20260619 18:58:16.357525 2528 heartbeater.cc:503] Master 127.15.91.62:39205 was elected leader, sending a full tablet report...
I20260619 18:58:16.358616 2549 ts_tablet_manager.cc:1428] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: Time spent starting tablet: real 0.016s user 0.007s sys 0.010s
I20260619 18:58:16.360531 2548 ts_tablet_manager.cc:1428] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
W20260619 18:58:16.398928 2267 tablet.cc:2369] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:58:16.434533 2398 tablet.cc:2369] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:58:16.440763 2554 raft_consensus.cc:482] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:58:16.441170 2554 raft_consensus.cc:504] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06148076a5a04e8d80d886905b616f75" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33111 } } peers { permanent_uuid: "6f218cd51c254b87b5f7d760417cfd05" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 45333 } } peers { permanent_uuid: "5e43a09c74c3414d9546f4ad1372b32a" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41505 } }
I20260619 18:58:16.443253 2554 leader_election.cc:288] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f218cd51c254b87b5f7d760417cfd05 (127.15.91.2:45333), 5e43a09c74c3414d9546f4ad1372b32a (127.15.91.1:41505)
I20260619 18:58:16.449858 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.443634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:41505 (local address 127.15.91.3:33629)
0619 18:58:16.444033 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:16.444057 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:16.444243 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:16.444644 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:16.444649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:16.444676 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:16.444935 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.444948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.446042 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.446048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.447678 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.447694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.448982 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.448989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.449196 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.449223 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:58:16.449351 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 18:58:16.449526 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":52}
I20260619 18:58:16.449826 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.443919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:33629 (local address 127.15.91.1:41505)
0619 18:58:16.444122 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:16.444128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:16.444150 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:58:16.444424 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:16.444429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:16.444506 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:16.444619 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:16.445145 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.445876 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.447907 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.448791 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.448962 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.449073 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:16.449388 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 18:58:16.449540 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:58:16.450698 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.443807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:58129 (local address 127.15.91.2:45333)
0619 18:58:16.444036 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:16.444045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:16.444836 (+ 791us) server_negotiation.cc:407] Connection header received
0619 18:58:16.445126 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:16.445131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:16.445249 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:16.445306 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:16.445984 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.446870 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.448826 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.449679 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.449845 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.449952 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:16.450334 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 18:58:16.450481 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 18:58:16.450918 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.443760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:45333 (local address 127.15.91.3:58129)
0619 18:58:16.444701 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:16.444722 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:16.444910 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:16.445462 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:16.445465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:16.445486 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:16.445797 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.445805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.447051 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.447062 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.448583 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:16.448599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.449895 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.449902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:16.450110 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.450160 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 18:58:16.450260 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 18:58:16.450451 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":25,"thread_start_us":558,"threads_started":1}
I20260619 18:58:16.451486 2221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cf7ccecc7043dfb84d73385bea5c57" candidate_uuid: "06148076a5a04e8d80d886905b616f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e43a09c74c3414d9546f4ad1372b32a" is_pre_election: true
I20260619 18:58:16.452198 2221 raft_consensus.cc:2454] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06148076a5a04e8d80d886905b616f75 in term 0.
I20260619 18:58:16.452390 2352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cf7ccecc7043dfb84d73385bea5c57" candidate_uuid: "06148076a5a04e8d80d886905b616f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f218cd51c254b87b5f7d760417cfd05" is_pre_election: true
I20260619 18:58:16.453052 2352 raft_consensus.cc:2454] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06148076a5a04e8d80d886905b616f75 in term 0.
I20260619 18:58:16.453338 2418 leader_election.cc:302] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06148076a5a04e8d80d886905b616f75, 5e43a09c74c3414d9546f4ad1372b32a; no voters:
I20260619 18:58:16.454174 2554 raft_consensus.cc:2781] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:58:16.454389 2554 raft_consensus.cc:482] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:58:16.454509 2554 raft_consensus.cc:3037] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:16.460039 2554 raft_consensus.cc:504] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06148076a5a04e8d80d886905b616f75" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33111 } } peers { permanent_uuid: "6f218cd51c254b87b5f7d760417cfd05" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 45333 } } peers { permanent_uuid: "5e43a09c74c3414d9546f4ad1372b32a" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41505 } }
I20260619 18:58:16.461153 2554 leader_election.cc:288] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [CANDIDATE]: Term 1 election: Requested vote from peers 6f218cd51c254b87b5f7d760417cfd05 (127.15.91.2:45333), 5e43a09c74c3414d9546f4ad1372b32a (127.15.91.1:41505)
W20260619 18:58:16.461787 2529 tablet.cc:2369] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:58:16.461839 2352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cf7ccecc7043dfb84d73385bea5c57" candidate_uuid: "06148076a5a04e8d80d886905b616f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f218cd51c254b87b5f7d760417cfd05"
I20260619 18:58:16.462113 2352 raft_consensus.cc:3037] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:16.462158 2221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cf7ccecc7043dfb84d73385bea5c57" candidate_uuid: "06148076a5a04e8d80d886905b616f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e43a09c74c3414d9546f4ad1372b32a"
I20260619 18:58:16.462463 2221 raft_consensus.cc:3037] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:16.464883 2352 raft_consensus.cc:2454] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06148076a5a04e8d80d886905b616f75 in term 1.
I20260619 18:58:16.465451 2221 raft_consensus.cc:2454] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06148076a5a04e8d80d886905b616f75 in term 1.
I20260619 18:58:16.465646 2419 leader_election.cc:302] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06148076a5a04e8d80d886905b616f75, 6f218cd51c254b87b5f7d760417cfd05; no voters:
I20260619 18:58:16.466056 2554 raft_consensus.cc:2781] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:16.466607 2554 raft_consensus.cc:686] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 1 LEADER]: Becoming Leader. State: Replica: 06148076a5a04e8d80d886905b616f75, State: Running, Role: LEADER
I20260619 18:58:16.467316 2554 consensus_queue.cc:237] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06148076a5a04e8d80d886905b616f75" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33111 } } peers { permanent_uuid: "6f218cd51c254b87b5f7d760417cfd05" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 45333 } } peers { permanent_uuid: "5e43a09c74c3414d9546f4ad1372b32a" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41505 } }
I20260619 18:58:16.473070 2078 catalog_manager.cc:5495] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06148076a5a04e8d80d886905b616f75 (127.15.91.3). New cstate: current_term: 1 leader_uuid: "06148076a5a04e8d80d886905b616f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06148076a5a04e8d80d886905b616f75" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 33111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f218cd51c254b87b5f7d760417cfd05" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 45333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e43a09c74c3414d9546f4ad1372b32a" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41505 } health_report { overall_health: UNKNOWN } } }
I20260619 18:58:16.496850 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.487943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.15.91.3:33111)
0619 18:58:16.488303 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:16.488308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:16.488329 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:58:16.488569 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:16.488573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:16.488662 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:16.488811 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:58:16.489489 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.490526 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.493057 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.493731 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.493891 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.494674 (+ 783us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:58:16.495670 (+ 996us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:58:16.495982 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:16.496455 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 18:58:16.496507 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234}
I20260619 18:58:16.507980 2352 raft_consensus.cc:1261] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05 [term 1 FOLLOWER]: Refusing update from remote peer 06148076a5a04e8d80d886905b616f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:16.508065 2221 raft_consensus.cc:1261] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a [term 1 FOLLOWER]: Refusing update from remote peer 06148076a5a04e8d80d886905b616f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:16.509320 2557 consensus_queue.cc:1035] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e43a09c74c3414d9546f4ad1372b32a" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:16.509841 2554 consensus_queue.cc:1035] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f218cd51c254b87b5f7d760417cfd05" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 45333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:16.524705 2560 mvcc.cc:205] Tried to move back new op lower bound from 7298643953684221952 to 7298643953534730240. Current Snapshot: MvccSnapshot[applied={T|T < 7298643953684221952}]
I20260619 18:58:16.545425 2079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:16.545770 2079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:16.550379 2079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e7e6d874bf94310a02ff87632f1edc7: Sending DeleteTablet for 3 replicas of tablet d0cf7ccecc7043dfb84d73385bea5c57
I20260619 18:58:16.551862 2332 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cf7ccecc7043dfb84d73385bea5c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:16 UTC) from {username='slave'} at 127.0.0.1:59672
I20260619 18:58:16.551863 2201 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cf7ccecc7043dfb84d73385bea5c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:16 UTC) from {username='slave'} at 127.0.0.1:55102
I20260619 18:58:16.551863 2463 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cf7ccecc7043dfb84d73385bea5c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:16 UTC) from {username='slave'} at 127.0.0.1:48190
I20260619 18:58:16.553128 2570 tablet_replica.cc:330] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: stopping tablet replica
I20260619 18:58:16.553449 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 2138
I20260619 18:58:16.553668 2570 raft_consensus.cc:2229] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:58:16.554450 2570 raft_consensus.cc:2258] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:16.555425 2569 tablet_replica.cc:330] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: stopping tablet replica
I20260619 18:58:16.555969 2569 raft_consensus.cc:2229] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:58:16.556566 2569 raft_consensus.cc:2258] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:16.557507 2571 tablet_replica.cc:330] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: stopping tablet replica
I20260619 18:58:16.557989 2571 raft_consensus.cc:2229] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:58:16.558638 2571 raft_consensus.cc:2258] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:16.559803 2569 ts_tablet_manager.cc:1905] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:16.561105 2571 ts_tablet_manager.cc:1905] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:16.562218 2570 ts_tablet_manager.cc:1905] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:16.568625 2569 ts_tablet_manager.cc:1918] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:16.568975 2569 log.cc:1196] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/d0cf7ccecc7043dfb84d73385bea5c57
I20260619 18:58:16.569195 2571 ts_tablet_manager.cc:1918] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:16.569406 2571 log.cc:1196] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/d0cf7ccecc7043dfb84d73385bea5c57
I20260619 18:58:16.569608 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.555752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.15.91.1:41505)
0619 18:58:16.555936 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:16.555944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:16.555975 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:58:16.556265 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:16.556272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:16.556407 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:16.556500 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:16.558028 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.559193 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.561267 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.562002 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.562140 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.567390 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:16.567406 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:16.567428 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:16.568346 (+ 918us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:16.568434 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:16.568469 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:16.568609 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:16.568986 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 18:58:16.569139 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:58:16.570111 2571 ts_tablet_manager.cc:1939] T d0cf7ccecc7043dfb84d73385bea5c57 P 6f218cd51c254b87b5f7d760417cfd05: Deleting consensus metadata
I20260619 18:58:16.570451 2569 ts_tablet_manager.cc:1939] T d0cf7ccecc7043dfb84d73385bea5c57 P 5e43a09c74c3414d9546f4ad1372b32a: Deleting consensus metadata
I20260619 18:58:16.570485 2570 ts_tablet_manager.cc:1918] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:16.570686 2570 log.cc:1196] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/d0cf7ccecc7043dfb84d73385bea5c57
I20260619 18:58:16.571101 2261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:16.572422 2066 catalog_manager.cc:4841] TS 6f218cd51c254b87b5f7d760417cfd05 (127.15.91.2:45333): tablet d0cf7ccecc7043dfb84d73385bea5c57 (table test-table [id=7606101615f84292876d344eb3223bf7]) successfully deleted
I20260619 18:58:16.573500 2570 ts_tablet_manager.cc:1939] T d0cf7ccecc7043dfb84d73385bea5c57 P 06148076a5a04e8d80d886905b616f75: Deleting consensus metadata
I20260619 18:58:16.575802 2064 catalog_manager.cc:4841] TS 06148076a5a04e8d80d886905b616f75 (127.15.91.3:33111): tablet d0cf7ccecc7043dfb84d73385bea5c57 (table test-table [id=7606101615f84292876d344eb3223bf7]) successfully deleted
I20260619 18:58:16.672576 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 2138
I20260619 18:58:16.672791 2065 catalog_manager.cc:4841] TS 5e43a09c74c3414d9546f4ad1372b32a (127.15.91.1:41505): tablet d0cf7ccecc7043dfb84d73385bea5c57 (table test-table [id=7606101615f84292876d344eb3223bf7]) successfully deleted
I20260619 18:58:16.696930 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 2270
I20260619 18:58:16.707891 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.697902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.15.91.2:45333)
0619 18:58:16.698101 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:16.698110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:16.698207 (+ 97us) server_negotiation.cc:407] Connection header received
0619 18:58:16.698395 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:16.698402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:16.698533 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:16.698672 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:16.699245 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.700546 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.702793 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.703518 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.703733 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.705524 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:16.705540 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:16.705575 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:16.706644 (+ 1069us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:16.706798 (+ 154us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:16.706839 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:16.707015 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:16.707264 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 18:58:16.707417 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:58:16.708743 2392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:16.811583 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 2270
I20260619 18:58:16.839385 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 2401
I20260619 18:58:16.847915 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:16.840498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.15.91.3:33111)
0619 18:58:16.840742 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:16.840747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:16.840801 (+ 54us) server_negotiation.cc:407] Connection header received
0619 18:58:16.840947 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:16.840953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:16.841056 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:16.841207 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:16.841656 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.842526 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.844026 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:16.844510 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:16.844703 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:16.846125 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:16.846139 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:16.846157 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:16.846969 (+ 812us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:16.847033 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:16.847059 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:16.847207 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:16.847390 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 18:58:16.847528 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:58:16.848788 2523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:16.978549 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 2401
I20260619 18:58:16.999312 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 2048
I20260619 18:58:17.000696 2109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:17.125214 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 2048
2026-06-19T18:58:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3240 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:17 dist-test-slave-dm3f krb5kdc[2579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:17 dist-test-slave-dm3f krb5kdc[2579](info): set up 2 sockets
Jun 19 18:58:17 dist-test-slave-dm3f krb5kdc[2579](info): commencing operation
krb5kdc: starting...
W20260619 18:58:18.267305 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:18 dist-test-slave-dm3f krb5kdc[2579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:58:18.537400 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:36047
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:36047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:18.903100 2592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:18.903434 2592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:18.903532 2592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:18.911500 2592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:18.911643 2592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:18.911692 2592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:18.911744 2592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:18.911791 2592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:18.924315 2592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:36047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:36047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:18.925729 2592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:18.927229 2592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:18.940757 2597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:18.940665 2598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:18.940690 2600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:18.941576 2592 server_base.cc:1034] running on GCE node
I20260619 18:58:18.942556 2592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:18.944410 2592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:18.945734 2592 hybrid_clock.cc:648] HybridClock initialized: now 1781895498945673 us; error 56 us; skew 500 ppm
I20260619 18:58:18.946220 2592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:18 dist-test-slave-dm3f krb5kdc[2579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895498, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:18.950162 2592 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:58:18.952276 2592 webserver.cc:460] Webserver started at http://127.15.91.62:34077/ using document root <none> and password file <none>
I20260619 18:58:18.953166 2592 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:18.953361 2592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:18.953732 2592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:18.957566 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "feba3802dad24fb288959b7e9bfa7446"
format_stamp: "Formatted at 2026-06-19 18:58:18 on dist-test-slave-dm3f"
server_key: "e15b63f56c48daaaabf7872fe47feb20"
server_key_iv: "7081b2dd01e1537fe658a38191bd93c1"
server_key_version: "encryptionkey@0"
I20260619 18:58:18.958612 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "feba3802dad24fb288959b7e9bfa7446"
format_stamp: "Formatted at 2026-06-19 18:58:18 on dist-test-slave-dm3f"
server_key: "e15b63f56c48daaaabf7872fe47feb20"
server_key_iv: "7081b2dd01e1537fe658a38191bd93c1"
server_key_version: "encryptionkey@0"
I20260619 18:58:18.965324 2592 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:58:18.969957 2607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:18.971817 2592 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:58:18.972102 2592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "feba3802dad24fb288959b7e9bfa7446"
format_stamp: "Formatted at 2026-06-19 18:58:18 on dist-test-slave-dm3f"
server_key: "e15b63f56c48daaaabf7872fe47feb20"
server_key_iv: "7081b2dd01e1537fe658a38191bd93c1"
server_key_version: "encryptionkey@0"
I20260619 18:58:18.972384 2592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:18.997958 2592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:19.002794 2592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:19.003144 2592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:19.024466 2592 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:36047
I20260619 18:58:19.024485 2658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:36047 every 8 connection(s)
I20260619 18:58:19.026172 2592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:19.030740 2659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:19.036157 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 2592
I20260619 18:58:19.036427 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:19.036895 15724 external_mini_cluster.cc:1428] Setting key cb7149df4662f08081ddad05ce55c10a
I20260619 18:58:19.043635 2659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446: Bootstrap starting.
I20260619 18:58:19.048134 2659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:19.049575 2659 log.cc:824] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:19.053234 2659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446: No bootstrap required, opened a new log
Jun 19 18:58:19 dist-test-slave-dm3f krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:19.062506 2659 raft_consensus.cc:348] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feba3802dad24fb288959b7e9bfa7446" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36047 } }
I20260619 18:58:19.063200 2659 raft_consensus.cc:374] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:19.063319 2659 raft_consensus.cc:729] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feba3802dad24fb288959b7e9bfa7446, State: Initialized, Role: FOLLOWER
I20260619 18:58:19.063997 2659 consensus_queue.cc:260] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feba3802dad24fb288959b7e9bfa7446" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36047 } }
I20260619 18:58:19.064293 2659 raft_consensus.cc:388] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:19.064452 2659 raft_consensus.cc:482] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:19.064670 2659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:19.064668 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:19.039110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.15.91.62:36047)
0619 18:58:19.040061 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:19.040075 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:19.040129 (+ 54us) server_negotiation.cc:407] Connection header received
0619 18:58:19.042109 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:19.042139 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:19.042636 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:19.043153 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:19.045263 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:19.046902 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:19.049045 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:19.050013 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:19.050280 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:19.055329 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:19.055341 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:19.055359 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:19.055412 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:19.059251 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:19.060245 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:19.060253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:19.060261 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:19.060485 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:19.060953 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:19.060964 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:19.060966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:19.061655 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:19.062002 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:19.062530 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 18:58:19.063177 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":457,"threads_started":1}
I20260619 18:58:19.068010 2659 raft_consensus.cc:504] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feba3802dad24fb288959b7e9bfa7446" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36047 } }
I20260619 18:58:19.068742 2659 leader_election.cc:302] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feba3802dad24fb288959b7e9bfa7446; no voters:
I20260619 18:58:19.069352 2659 leader_election.cc:288] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:19.069525 2664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:19.070299 2664 raft_consensus.cc:686] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [term 1 LEADER]: Becoming Leader. State: Replica: feba3802dad24fb288959b7e9bfa7446, State: Running, Role: LEADER
I20260619 18:58:19.071019 2664 consensus_queue.cc:237] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feba3802dad24fb288959b7e9bfa7446" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36047 } }
I20260619 18:58:19.071746 2659 sys_catalog.cc:564] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:19.074414 2666 sys_catalog.cc:455] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feba3802dad24fb288959b7e9bfa7446. Latest consensus state: current_term: 1 leader_uuid: "feba3802dad24fb288959b7e9bfa7446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feba3802dad24fb288959b7e9bfa7446" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36047 } } }
I20260619 18:58:19.075069 2666 sys_catalog.cc:458] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:19.075757 2665 sys_catalog.cc:455] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feba3802dad24fb288959b7e9bfa7446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feba3802dad24fb288959b7e9bfa7446" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 36047 } } }
I20260619 18:58:19.076128 2665 sys_catalog.cc:458] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:19.078980 2671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:19.085237 2671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:19.097318 2671 catalog_manager.cc:1347] Generated new cluster ID: 81a06f358e994865b00037bebfa2c1ea
I20260619 18:58:19.097502 2671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:19.129837 2671 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:19.131443 2671 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:19.143985 2671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446: Generated new TSK 0
I20260619 18:58:19.145056 2671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.91.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
Entry for principal kudu/127.15.91.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab.
I20260619 18:58:19.215016 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36047
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:19.595392 2685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:19.595715 2685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:19.595877 2685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:19.603792 2685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:19.603930 2685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:19.604115 2685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:58:19.614619 2685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.1.keytab
--principal=kudu/127.15.91.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:19.616109 2685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:19.617654 2685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:19.633062 2690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:19.633443 2693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:19.633476 2691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:19.634629 2685 server_base.cc:1034] running on GCE node
I20260619 18:58:19.635422 2685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:19.637320 2685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:19.638640 2685 hybrid_clock.cc:648] HybridClock initialized: now 1781895499638569 us; error 70 us; skew 500 ppm
I20260619 18:58:19.639570 2685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:19 dist-test-slave-dm3f krb5kdc[2579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:19.643745 2685 init.cc:377] Logged in from keytab as kudu/127.15.91.1@KRBTEST.COM (short username kudu)
I20260619 18:58:19.645673 2685 webserver.cc:460] Webserver started at http://127.15.91.1:32979/ using document root <none> and password file <none>
I20260619 18:58:19.646518 2685 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:19.646695 2685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:19.647125 2685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:19.650585 2685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "28333353307340039675050c246ac39e"
format_stamp: "Formatted at 2026-06-19 18:58:19 on dist-test-slave-dm3f"
server_key: "2e357a9b14d9bc7b866f183061770967"
server_key_iv: "1afe2f1975f7e7371c2387247c38af0e"
server_key_version: "encryptionkey@0"
I20260619 18:58:19.651922 2685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "28333353307340039675050c246ac39e"
format_stamp: "Formatted at 2026-06-19 18:58:19 on dist-test-slave-dm3f"
server_key: "2e357a9b14d9bc7b866f183061770967"
server_key_iv: "1afe2f1975f7e7371c2387247c38af0e"
server_key_version: "encryptionkey@0"
I20260619 18:58:19.658694 2685 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:58:19.663605 2700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:19.665381 2685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:58:19.665649 2685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "28333353307340039675050c246ac39e"
format_stamp: "Formatted at 2026-06-19 18:58:19 on dist-test-slave-dm3f"
server_key: "2e357a9b14d9bc7b866f183061770967"
server_key_iv: "1afe2f1975f7e7371c2387247c38af0e"
server_key_version: "encryptionkey@0"
I20260619 18:58:19.665916 2685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:19.684958 2685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:19.690071 2685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:19.690456 2685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:19.691720 2685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:19.693406 2685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:19.693548 2685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:19.693732 2685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:19.693811 2685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:19.732196 2685 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:44223
I20260619 18:58:19.732239 2813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:44223 every 8 connection(s)
I20260619 18:58:19.733712 2685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:58:19.742278 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 2685
I20260619 18:58:19.742520 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:58:19.743093 15724 external_mini_cluster.cc:1428] Setting key 041f50b13ef39651ac45321a4b5d234d
Jun 19 18:58:19 dist-test-slave-dm3f krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.1@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:19.759213 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:19.737530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:57659 (local address 127.15.91.62:36047)
0619 18:58:19.738055 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:19.738060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:19.739380 (+ 1320us) server_negotiation.cc:407] Connection header received
0619 18:58:19.741516 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:19.741523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:19.741649 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:19.741788 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:19.744007 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:19.745044 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:19.746980 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:19.747422 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:19.747603 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:19.752054 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:19.752060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:19.752064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:19.752105 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:19.755270 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:19.756297 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:19.756303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:19.756305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:19.756452 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:19.756999 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:19.757003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:19.757005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:19.757482 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:19.757648 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:19.758498 (+ 850us) server_negotiation.cc:299] Negotiation successful
0619 18:58:19.758814 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":273,"threads_started":1}
I20260619 18:58:19.760352 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:19.738192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:36047 (local address 127.15.91.1:57659)
0619 18:58:19.739165 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:19.739221 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:19.741125 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:19.742125 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:19.742138 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:19.742997 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:19.743761 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:19.743785 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:19.745258 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:19.745264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:19.746683 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:19.746706 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:19.747741 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:19.747749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:19.748028 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:19.748841 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:58:19.748868 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:19.751694 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:19.755507 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:19.755520 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:19.755554 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:19.756105 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:19.756599 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:19.756605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:19.756609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:19.756849 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:19.757627 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:19.757639 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:19.758065 (+ 426us) client_negotiation.cc:769] Sending connection context
0619 18:58:19.758375 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 18:58:19.759050 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":346,"threads_started":1}
I20260619 18:58:19.762329 2814 heartbeater.cc:347] Connected to a master server at 127.15.91.62:36047
I20260619 18:58:19.762759 2814 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:19.763653 2814 heartbeater.cc:511] Master 127.15.91.62:36047 requested a full tablet report, sending...
I20260619 18:58:19.766826 2624 ts_manager.cc:194] Registered new tserver with Master: 28333353307340039675050c246ac39e (127.15.91.1:44223)
WARNING: no policy specified for kudu/127.15.91.2@KRBTEST.COM; defaulting to no policy
I20260619 18:58:19.769338 2624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.1@KRBTEST.COM'} at 127.15.91.1:57659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
Entry for principal kudu/127.15.91.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab.
I20260619 18:58:19.802906 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36047
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:20.172850 2820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:20.173218 2820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:20.173347 2820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:20.181552 2820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:20.181700 2820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:20.181823 2820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:58:20.192735 2820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.2.keytab
--principal=kudu/127.15.91.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:20.194208 2820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:20.195765 2820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:20.211048 2825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:20.211076 2826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:20.211009 2828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:20.212533 2820 server_base.cc:1034] running on GCE node
I20260619 18:58:20.213265 2820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:20.214409 2820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:20.215641 2820 hybrid_clock.cc:648] HybridClock initialized: now 1781895500215553 us; error 87 us; skew 500 ppm
I20260619 18:58:20.215981 2820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:20 dist-test-slave-dm3f krb5kdc[2579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895500, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:20.219796 2820 init.cc:377] Logged in from keytab as kudu/127.15.91.2@KRBTEST.COM (short username kudu)
I20260619 18:58:20.221581 2820 webserver.cc:460] Webserver started at http://127.15.91.2:35789/ using document root <none> and password file <none>
I20260619 18:58:20.222429 2820 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:20.222610 2820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:20.223003 2820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:20.226311 2820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "86f4b9a668c4423283f6363a32260883"
format_stamp: "Formatted at 2026-06-19 18:58:20 on dist-test-slave-dm3f"
server_key: "a002dcadda05c4ec7e304fbd52efbb0d"
server_key_iv: "c452c76b256304297cef5372d8b383c9"
server_key_version: "encryptionkey@0"
I20260619 18:58:20.227579 2820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "86f4b9a668c4423283f6363a32260883"
format_stamp: "Formatted at 2026-06-19 18:58:20 on dist-test-slave-dm3f"
server_key: "a002dcadda05c4ec7e304fbd52efbb0d"
server_key_iv: "c452c76b256304297cef5372d8b383c9"
server_key_version: "encryptionkey@0"
I20260619 18:58:20.234292 2820 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:58:20.238618 2835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:20.241102 2820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:20.241381 2820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "86f4b9a668c4423283f6363a32260883"
format_stamp: "Formatted at 2026-06-19 18:58:20 on dist-test-slave-dm3f"
server_key: "a002dcadda05c4ec7e304fbd52efbb0d"
server_key_iv: "c452c76b256304297cef5372d8b383c9"
server_key_version: "encryptionkey@0"
I20260619 18:58:20.241668 2820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:20.282407 2820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:20.287707 2820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:20.288080 2820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:20.289220 2820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:20.290879 2820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:20.291010 2820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:20.291229 2820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:20.291320 2820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:20.330642 2820 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:39053
I20260619 18:58:20.330869 2947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:39053 every 8 connection(s)
I20260619 18:58:20.332244 2820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:58:20.342653 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 2820
I20260619 18:58:20.342909 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:58:20.343379 15724 external_mini_cluster.cc:1428] Setting key 8a28f687f02feec6541a659778c59127
Jun 19 18:58:20 dist-test-slave-dm3f krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895500, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.2@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:20.355672 2950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:20.335259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:43111 (local address 127.15.91.62:36047)
0619 18:58:20.335764 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:20.335769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:20.336935 (+ 1166us) server_negotiation.cc:407] Connection header received
0619 18:58:20.339300 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:20.339306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:20.339420 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:20.339552 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:20.341396 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.342150 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.344183 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.344717 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.344880 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:20.348874 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:20.348880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:20.348884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:20.348923 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:20.351854 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:20.352852 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:20.352857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:20.352859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:20.353002 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:20.353535 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:20.353539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:20.353541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:20.353861 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:20.354065 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:20.354983 (+ 918us) server_negotiation.cc:299] Negotiation successful
0619 18:58:20.355291 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":251,"threads_started":1}
I20260619 18:58:20.356812 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:20.335915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:36047 (local address 127.15.91.2:43111)
0619 18:58:20.336727 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:20.336772 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:20.338942 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:20.339864 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:20.339875 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:20.340498 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:20.341160 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:20.341182 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.342352 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.342357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:20.343980 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:20.343997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.344992 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.345000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:20.345284 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:20.346024 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:58:20.346051 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:20.348542 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:20.352087 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:20.352099 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:20.352130 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:20.352670 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:20.353140 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:20.353145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:20.353149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:20.353383 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:20.354018 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:20.354028 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:20.354445 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 18:58:20.354846 (+ 401us) client_negotiation.cc:240] Negotiation successful
0619 18:58:20.355476 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":321,"threads_started":1}
I20260619 18:58:20.358633 2948 heartbeater.cc:347] Connected to a master server at 127.15.91.62:36047
I20260619 18:58:20.359083 2948 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:20.360178 2948 heartbeater.cc:511] Master 127.15.91.62:36047 requested a full tablet report, sending...
I20260619 18:58:20.362283 2624 ts_manager.cc:194] Registered new tserver with Master: 86f4b9a668c4423283f6363a32260883 (127.15.91.2:39053)
I20260619 18:58:20.363339 2624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.2@KRBTEST.COM'} at 127.15.91.2:43111
WARNING: no policy specified for kudu/127.15.91.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
Entry for principal kudu/127.15.91.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab.
I20260619 18:58:20.402194 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36047
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:20.768559 2954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:20.768875 2954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:20.769053 2954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 18:58:20.773801 2814 heartbeater.cc:503] Master 127.15.91.62:36047 was elected leader, sending a full tablet report...
W20260619 18:58:20.778201 2954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:20.778389 2954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:20.778575 2954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:58:20.789237 2954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.3.keytab
--principal=kudu/127.15.91.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:36047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:20.790758 2954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:20.792217 2954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:20.807075 2962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:20.807133 2960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:20.807348 2959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:20.808777 2954 server_base.cc:1034] running on GCE node
I20260619 18:58:20.809587 2954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:20.810851 2954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:20.812124 2954 hybrid_clock.cc:648] HybridClock initialized: now 1781895500812063 us; error 64 us; skew 500 ppm
I20260619 18:58:20.812546 2954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:20 dist-test-slave-dm3f krb5kdc[2579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895500, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:20.816442 2954 init.cc:377] Logged in from keytab as kudu/127.15.91.3@KRBTEST.COM (short username kudu)
I20260619 18:58:20.818336 2954 webserver.cc:460] Webserver started at http://127.15.91.3:42239/ using document root <none> and password file <none>
I20260619 18:58:20.819253 2954 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:20.819422 2954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:20.819811 2954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:20.823252 2954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "6d5b5ffc89124760849410690d48a190"
format_stamp: "Formatted at 2026-06-19 18:58:20 on dist-test-slave-dm3f"
server_key: "6824ee7224c3d5daa7f2531b7ca54406"
server_key_iv: "96c0b054e091a63836e8b14741b3ae48"
server_key_version: "encryptionkey@0"
I20260619 18:58:20.824235 2954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "6d5b5ffc89124760849410690d48a190"
format_stamp: "Formatted at 2026-06-19 18:58:20 on dist-test-slave-dm3f"
server_key: "6824ee7224c3d5daa7f2531b7ca54406"
server_key_iv: "96c0b054e091a63836e8b14741b3ae48"
server_key_version: "encryptionkey@0"
I20260619 18:58:20.830695 2954 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260619 18:58:20.835142 2969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:20.836850 2954 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:58:20.837095 2954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "6d5b5ffc89124760849410690d48a190"
format_stamp: "Formatted at 2026-06-19 18:58:20 on dist-test-slave-dm3f"
server_key: "6824ee7224c3d5daa7f2531b7ca54406"
server_key_iv: "96c0b054e091a63836e8b14741b3ae48"
server_key_version: "encryptionkey@0"
I20260619 18:58:20.837335 2954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:20.852147 2954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:20.857060 2954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:20.857412 2954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:20.858507 2954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:20.860280 2954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:20.860373 2954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:20.860553 2954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:20.860607 2954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:20.898463 2954 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:41811
I20260619 18:58:20.898514 3081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:41811 every 8 connection(s)
I20260619 18:58:20.900436 2954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:58:20.910665 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 2954
I20260619 18:58:20.910892 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:58:20.911463 15724 external_mini_cluster.cc:1428] Setting key 420ec4580ee9fff08dd87931568f6e2c
Jun 19 18:58:20 dist-test-slave-dm3f krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895500, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.3@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:20.924402 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:20.903462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:33501 (local address 127.15.91.62:36047)
0619 18:58:20.903968 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:20.903973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:20.904877 (+ 904us) server_negotiation.cc:407] Connection header received
0619 18:58:20.907024 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:20.907030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:20.907160 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:20.907293 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:20.909132 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.909921 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.912610 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.913302 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.913432 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:20.917940 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:20.917945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:20.917949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:20.917990 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:20.920859 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:20.921849 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:20.921854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:20.921856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:20.922002 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:20.922530 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:20.922534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:20.922536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:20.922930 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:20.923101 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:20.923877 (+ 776us) server_negotiation.cc:299] Negotiation successful
0619 18:58:20.924111 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":268,"threads_started":1}
I20260619 18:58:20.925732 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:20.903863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:36047 (local address 127.15.91.3:33501)
0619 18:58:20.904676 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:20.904738 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:20.906592 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:20.907592 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:20.907602 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:20.908213 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:20.908877 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:20.908900 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.910136 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.910140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:20.912345 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:20.912376 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.913616 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.913622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:20.914008 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:20.914849 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:58:20.914878 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:20.917611 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:20.921087 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:20.921097 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:20.921114 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:20.921684 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:20.922118 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:58:20.922129 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:58:20.922132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:58:20.922396 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:58:20.923064 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:20.923072 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:20.923485 (+ 413us) client_negotiation.cc:769] Sending connection context
0619 18:58:20.923750 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 18:58:20.924194 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":321,"threads_started":1}
I20260619 18:58:20.927861 3082 heartbeater.cc:347] Connected to a master server at 127.15.91.62:36047
I20260619 18:58:20.928293 3082 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:20.929104 3082 heartbeater.cc:511] Master 127.15.91.62:36047 requested a full tablet report, sending...
I20260619 18:58:20.930917 2624 ts_manager.cc:194] Registered new tserver with Master: 6d5b5ffc89124760849410690d48a190 (127.15.91.3:41811)
I20260619 18:58:20.931779 2624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.91.3@KRBTEST.COM'} at 127.15.91.3:33501
I20260619 18:58:20.943567 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:58:20.963016 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:20.948306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.15.91.62:36047)
0619 18:58:20.948584 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:20.948590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:20.948919 (+ 329us) server_negotiation.cc:407] Connection header received
0619 18:58:20.949073 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:20.949078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:20.949169 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:20.949288 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:20.950936 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.951721 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.953257 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.953681 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.953800 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:20.956690 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:20.956696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:20.956699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:20.956743 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:20.959785 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:20.960740 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:20.960745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:20.960748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:20.960917 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:20.961358 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:20.961364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:20.961366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:20.961761 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:20.961966 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:20.962353 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 18:58:20.962562 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:58:20.968406 2624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:58:20.976311 2624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:58:21.003964 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:20.991060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.15.91.2:39053)
0619 18:58:20.991801 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:20.991815 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:20.992614 (+ 799us) server_negotiation.cc:407] Connection header received
0619 18:58:20.994296 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:20.994306 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:20.994540 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:20.994758 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:20.995824 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.997488 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.001374 (+ 3886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.002409 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.002598 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.002885 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:21.003381 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 18:58:21.003585 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":440,"threads_started":1}
I20260619 18:58:21.004155 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:20.991197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:41811 (local address 127.0.0.1:33350)
0619 18:58:20.992390 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:20.992675 (+ 285us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:20.992861 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:20.994828 (+ 1967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:20.994839 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:20.994873 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:20.995385 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:20.995403 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.997713 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.997717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:21.000695 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:21.000724 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.003009 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.003017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:21.003244 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.003282 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 18:58:21.003360 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:58:21.003602 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":173,"thread_start_us":819,"threads_started":1}
I20260619 18:58:21.004773 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:20.993650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51824 (local address 127.15.91.1:44223)
0619 18:58:20.995084 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:20.995091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:20.995114 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:58:20.995192 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:20.995197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:20.995420 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:20.996396 (+ 976us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:20.996700 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.998014 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.000972 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.002557 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.003447 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.003689 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:21.003771 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 18:58:21.003952 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1266,"thread_start_us":1093,"threads_started":1}
I20260619 18:58:21.005019 3098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:20.991815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.15.91.3:41811)
0619 18:58:20.993690 (+ 1875us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:20.993703 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:20.993746 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:58:20.993869 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:20.993881 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:20.994193 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:20.994424 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:20.995679 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.997512 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.000970 (+ 3458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.002557 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.002838 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.003095 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:21.004420 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0619 18:58:21.004653 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1636,"thread_start_us":1579,"threads_started":1}
I20260619 18:58:21.004177 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:20.993328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:44223 (local address 127.0.0.1:51824)
0619 18:58:20.994185 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:20.994208 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:20.994378 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:20.995661 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:20.995666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:20.995708 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:20.996157 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:20.996170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.998195 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.998201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:21.000763 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:21.000773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.002881 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.002888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:21.003087 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.003132 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 18:58:21.003237 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 18:58:21.003602 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":100,"thread_start_us":462,"threads_started":1}
I20260619 18:58:21.006628 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:20.991051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:39053 (local address 127.0.0.1:50882)
0619 18:58:20.992390 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:20.992472 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:20.994057 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:20.995076 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:20.995079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:20.995101 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:20.995601 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:20.995614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:20.997675 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:20.997682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:21.001154 (+ 3472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:21.001169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.002658 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.002666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:21.003014 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.003106 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 18:58:21.003269 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 18:58:21.004321 (+ 1052us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":1780,"thread_start_us":942,"threads_started":1}
I20260619 18:58:21.008952 3017 tablet_service.cc:1467] Processing CreateTablet for tablet 26103012be924acb996660e417bd78f5 (DEFAULT_TABLE table=test-table [id=02dd33bc3f9f498fad4f4b72e323d6da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:21.010552 3017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26103012be924acb996660e417bd78f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:21.010141 2883 tablet_service.cc:1467] Processing CreateTablet for tablet 26103012be924acb996660e417bd78f5 (DEFAULT_TABLE table=test-table [id=02dd33bc3f9f498fad4f4b72e323d6da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:21.011760 2883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26103012be924acb996660e417bd78f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:21.014204 2748 tablet_service.cc:1467] Processing CreateTablet for tablet 26103012be924acb996660e417bd78f5 (DEFAULT_TABLE table=test-table [id=02dd33bc3f9f498fad4f4b72e323d6da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:21.016137 2748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26103012be924acb996660e417bd78f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:21.028812 3101 tablet_bootstrap.cc:492] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: Bootstrap starting.
I20260619 18:58:21.032954 3101 tablet_bootstrap.cc:654] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:21.034451 3101 log.cc:824] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:21.036705 3102 tablet_bootstrap.cc:492] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: Bootstrap starting.
I20260619 18:58:21.039623 3104 tablet_bootstrap.cc:492] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: Bootstrap starting.
I20260619 18:58:21.043181 3102 tablet_bootstrap.cc:654] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:21.043591 3104 tablet_bootstrap.cc:654] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:21.044750 3102 log.cc:824] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:21.045935 3104 log.cc:824] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:21.048502 3101 tablet_bootstrap.cc:492] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: No bootstrap required, opened a new log
I20260619 18:58:21.048501 3102 tablet_bootstrap.cc:492] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: No bootstrap required, opened a new log
I20260619 18:58:21.048847 3102 ts_tablet_manager.cc:1397] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20260619 18:58:21.048847 3101 ts_tablet_manager.cc:1397] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.009s
I20260619 18:58:21.052425 3104 tablet_bootstrap.cc:492] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: No bootstrap required, opened a new log
I20260619 18:58:21.052840 3104 ts_tablet_manager.cc:1397] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260619 18:58:21.061162 3101 raft_consensus.cc:348] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f4b9a668c4423283f6363a32260883" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39053 } } peers { permanent_uuid: "6d5b5ffc89124760849410690d48a190" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 41811 } } peers { permanent_uuid: "28333353307340039675050c246ac39e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44223 } }
I20260619 18:58:21.061162 3102 raft_consensus.cc:348] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f4b9a668c4423283f6363a32260883" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39053 } } peers { permanent_uuid: "6d5b5ffc89124760849410690d48a190" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 41811 } } peers { permanent_uuid: "28333353307340039675050c246ac39e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44223 } }
I20260619 18:58:21.061324 3104 raft_consensus.cc:348] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f4b9a668c4423283f6363a32260883" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39053 } } peers { permanent_uuid: "6d5b5ffc89124760849410690d48a190" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 41811 } } peers { permanent_uuid: "28333353307340039675050c246ac39e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44223 } }
I20260619 18:58:21.061760 3102 raft_consensus.cc:374] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:21.061759 3101 raft_consensus.cc:374] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:21.061831 3104 raft_consensus.cc:374] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:21.061884 3102 raft_consensus.cc:729] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f4b9a668c4423283f6363a32260883, State: Initialized, Role: FOLLOWER
I20260619 18:58:21.061884 3101 raft_consensus.cc:729] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5b5ffc89124760849410690d48a190, State: Initialized, Role: FOLLOWER
I20260619 18:58:21.061934 3104 raft_consensus.cc:729] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28333353307340039675050c246ac39e, State: Initialized, Role: FOLLOWER
I20260619 18:58:21.062669 3101 consensus_queue.cc:260] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f4b9a668c4423283f6363a32260883" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39053 } } peers { permanent_uuid: "6d5b5ffc89124760849410690d48a190" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 41811 } } peers { permanent_uuid: "28333353307340039675050c246ac39e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44223 } }
I20260619 18:58:21.062669 3102 consensus_queue.cc:260] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f4b9a668c4423283f6363a32260883" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39053 } } peers { permanent_uuid: "6d5b5ffc89124760849410690d48a190" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 41811 } } peers { permanent_uuid: "28333353307340039675050c246ac39e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44223 } }
I20260619 18:58:21.062805 3104 consensus_queue.cc:260] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f4b9a668c4423283f6363a32260883" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39053 } } peers { permanent_uuid: "6d5b5ffc89124760849410690d48a190" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 41811 } } peers { permanent_uuid: "28333353307340039675050c246ac39e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44223 } }
I20260619 18:58:21.064886 3082 heartbeater.cc:503] Master 127.15.91.62:36047 was elected leader, sending a full tablet report...
I20260619 18:58:21.065105 3102 ts_tablet_manager.cc:1428] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260619 18:58:21.065361 3101 ts_tablet_manager.cc:1428] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: Time spent starting tablet: real 0.016s user 0.011s sys 0.006s
I20260619 18:58:21.066332 2948 heartbeater.cc:503] Master 127.15.91.62:36047 was elected leader, sending a full tablet report...
I20260619 18:58:21.067159 3104 ts_tablet_manager.cc:1428] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
W20260619 18:58:21.084517 2949 tablet.cc:2369] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:58:21.152370 3083 tablet.cc:2369] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:58:21.237365 2815 tablet.cc:2369] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:58:21.237617 3109 raft_consensus.cc:482] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:58:21.237923 3109 raft_consensus.cc:504] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f4b9a668c4423283f6363a32260883" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39053 } } peers { permanent_uuid: "6d5b5ffc89124760849410690d48a190" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 41811 } } peers { permanent_uuid: "28333353307340039675050c246ac39e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44223 } }
I20260619 18:58:21.239974 3109 leader_election.cc:288] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86f4b9a668c4423283f6363a32260883 (127.15.91.2:39053), 6d5b5ffc89124760849410690d48a190 (127.15.91.3:41811)
I20260619 18:58:21.248256 3098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:21.240532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:42053 (local address 127.15.91.3:41811)
0619 18:58:21.240800 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:21.240806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:21.241483 (+ 677us) server_negotiation.cc:407] Connection header received
0619 18:58:21.241759 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:21.241767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:21.241847 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:21.241951 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:21.242653 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.243558 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.245890 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.247077 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.247274 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.247412 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:21.247817 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 18:58:21.247967 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260619 18:58:21.248592 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:21.240575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:40161 (local address 127.15.91.2:39053)
0619 18:58:21.240799 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:21.240806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:21.242038 (+ 1232us) server_negotiation.cc:407] Connection header received
0619 18:58:21.242267 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:21.242272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:21.242366 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:21.242485 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:21.243128 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.243893 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.246177 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.247368 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.247563 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.247672 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:21.248035 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 18:58:21.248190 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:58:21.248687 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:21.240410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:41811 (local address 127.15.91.1:42053)
0619 18:58:21.241342 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:21.241362 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:21.241576 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:21.241988 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:21.241993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:21.242023 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:21.242481 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:21.242492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.243733 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.243739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:21.245601 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:21.245626 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.247393 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.247401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:21.247588 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.247621 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:58:21.247763 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 18:58:21.247929 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":5,"thread_start_us":559,"threads_started":1}
I20260619 18:58:21.248751 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:21.240859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:39053 (local address 127.15.91.1:40161)
0619 18:58:21.241898 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:21.241915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:21.242104 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:21.242505 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:21.242510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:21.242538 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:21.242950 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:21.242961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.244059 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.244065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:21.245903 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:21.245920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.247651 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.247659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:21.247855 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.247882 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:58:21.248003 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 18:58:21.248159 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":78,"thread_start_us":613,"threads_started":1}
I20260619 18:58:21.249953 3037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26103012be924acb996660e417bd78f5" candidate_uuid: "28333353307340039675050c246ac39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5b5ffc89124760849410690d48a190" is_pre_election: true
I20260619 18:58:21.249979 2903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26103012be924acb996660e417bd78f5" candidate_uuid: "28333353307340039675050c246ac39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f4b9a668c4423283f6363a32260883" is_pre_election: true
I20260619 18:58:21.250541 3037 raft_consensus.cc:2454] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28333353307340039675050c246ac39e in term 0.
I20260619 18:58:21.250545 2903 raft_consensus.cc:2454] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28333353307340039675050c246ac39e in term 0.
I20260619 18:58:21.251482 2702 leader_election.cc:302] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28333353307340039675050c246ac39e, 6d5b5ffc89124760849410690d48a190, 86f4b9a668c4423283f6363a32260883; no voters:
I20260619 18:58:21.252032 3109 raft_consensus.cc:2781] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:58:21.252213 3109 raft_consensus.cc:482] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:58:21.252323 3109 raft_consensus.cc:3037] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:21.254890 3109 raft_consensus.cc:504] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f4b9a668c4423283f6363a32260883" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39053 } } peers { permanent_uuid: "6d5b5ffc89124760849410690d48a190" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 41811 } } peers { permanent_uuid: "28333353307340039675050c246ac39e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44223 } }
I20260619 18:58:21.255705 3109 leader_election.cc:288] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [CANDIDATE]: Term 1 election: Requested vote from peers 86f4b9a668c4423283f6363a32260883 (127.15.91.2:39053), 6d5b5ffc89124760849410690d48a190 (127.15.91.3:41811)
I20260619 18:58:21.256346 3037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26103012be924acb996660e417bd78f5" candidate_uuid: "28333353307340039675050c246ac39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5b5ffc89124760849410690d48a190"
I20260619 18:58:21.256352 2903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26103012be924acb996660e417bd78f5" candidate_uuid: "28333353307340039675050c246ac39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f4b9a668c4423283f6363a32260883"
I20260619 18:58:21.256629 3037 raft_consensus.cc:3037] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:21.256605 2903 raft_consensus.cc:3037] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:21.259341 3037 raft_consensus.cc:2454] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28333353307340039675050c246ac39e in term 1.
I20260619 18:58:21.259440 2903 raft_consensus.cc:2454] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28333353307340039675050c246ac39e in term 1.
I20260619 18:58:21.259979 2701 leader_election.cc:302] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28333353307340039675050c246ac39e, 6d5b5ffc89124760849410690d48a190; no voters:
I20260619 18:58:21.260412 3109 raft_consensus.cc:2781] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:21.260953 3109 raft_consensus.cc:686] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 1 LEADER]: Becoming Leader. State: Replica: 28333353307340039675050c246ac39e, State: Running, Role: LEADER
I20260619 18:58:21.261593 3109 consensus_queue.cc:237] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f4b9a668c4423283f6363a32260883" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39053 } } peers { permanent_uuid: "6d5b5ffc89124760849410690d48a190" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 41811 } } peers { permanent_uuid: "28333353307340039675050c246ac39e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44223 } }
I20260619 18:58:21.266907 2624 catalog_manager.cc:5495] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28333353307340039675050c246ac39e (127.15.91.1). New cstate: current_term: 1 leader_uuid: "28333353307340039675050c246ac39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86f4b9a668c4423283f6363a32260883" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d5b5ffc89124760849410690d48a190" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 41811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28333353307340039675050c246ac39e" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 44223 } health_report { overall_health: HEALTHY } } }
I20260619 18:58:21.309721 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:21.303244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.15.91.1:44223)
0619 18:58:21.303441 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:21.303446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:21.303552 (+ 106us) server_negotiation.cc:407] Connection header received
0619 18:58:21.303763 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:21.303767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:21.303880 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:21.304032 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:58:21.304795 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.305537 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.307310 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.307770 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.307898 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.308519 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:58:21.309095 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:58:21.309234 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:21.309410 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 18:58:21.309494 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:58:21.319650 3037 raft_consensus.cc:1261] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190 [term 1 FOLLOWER]: Refusing update from remote peer 28333353307340039675050c246ac39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:21.319650 2903 raft_consensus.cc:1261] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883 [term 1 FOLLOWER]: Refusing update from remote peer 28333353307340039675050c246ac39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:21.321094 3112 consensus_queue.cc:1035] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5b5ffc89124760849410690d48a190" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 41811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:21.321576 3109 consensus_queue.cc:1035] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86f4b9a668c4423283f6363a32260883" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 39053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:21.337914 3115 mvcc.cc:205] Tried to move back new op lower bound from 7298643973392314368 to 7298643973171806208. Current Snapshot: MvccSnapshot[applied={T|T < 7298643973392314368}]
I20260619 18:58:21.346792 3117 mvcc.cc:205] Tried to move back new op lower bound from 7298643973392314368 to 7298643973171806208. Current Snapshot: MvccSnapshot[applied={T|T < 7298643973392314368}]
I20260619 18:58:21.366166 2624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:21.366513 2624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:21.371445 2624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P feba3802dad24fb288959b7e9bfa7446: Sending DeleteTablet for 3 replicas of tablet 26103012be924acb996660e417bd78f5
I20260619 18:58:21.373062 3017 tablet_service.cc:1514] Processing DeleteTablet for tablet 26103012be924acb996660e417bd78f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:21 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:33350
I20260619 18:58:21.373027 2883 tablet_service.cc:1514] Processing DeleteTablet for tablet 26103012be924acb996660e417bd78f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:21 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:50882
I20260619 18:58:21.373368 2748 tablet_service.cc:1514] Processing DeleteTablet for tablet 26103012be924acb996660e417bd78f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:21 UTC) from {username='kudu', principal='kudu/127.15.91.62@KRBTEST.COM'} at 127.0.0.1:51824
I20260619 18:58:21.376164 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 2685
I20260619 18:58:21.386178 3126 tablet_replica.cc:330] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: stopping tablet replica
I20260619 18:58:21.388496 3126 raft_consensus.cc:2229] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:58:21.391137 3126 raft_consensus.cc:2258] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:21.393903 3126 ts_tablet_manager.cc:1905] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:21.402669 3126 ts_tablet_manager.cc:1918] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:21.403100 3126 log.cc:1196] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/26103012be924acb996660e417bd78f5
I20260619 18:58:21.403826 3126 ts_tablet_manager.cc:1939] T 26103012be924acb996660e417bd78f5 P 6d5b5ffc89124760849410690d48a190: Deleting consensus metadata
I20260619 18:58:21.404692 3127 tablet_replica.cc:330] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: stopping tablet replica
I20260619 18:58:21.405400 3127 raft_consensus.cc:2229] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:58:21.406375 3127 raft_consensus.cc:2258] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:21.406508 2608 catalog_manager.cc:4841] TS 6d5b5ffc89124760849410690d48a190 (127.15.91.3:41811): tablet 26103012be924acb996660e417bd78f5 (table test-table [id=02dd33bc3f9f498fad4f4b72e323d6da]) successfully deleted
I20260619 18:58:21.410130 3127 ts_tablet_manager.cc:1905] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:21.411712 3125 tablet_replica.cc:330] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: stopping tablet replica
I20260619 18:58:21.412284 3125 raft_consensus.cc:2229] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:58:21.412909 3125 raft_consensus.cc:2258] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:21.416092 3125 ts_tablet_manager.cc:1905] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:58:21 dist-test-slave-dm3f krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.1@KRBTEST.COM
I20260619 18:58:21.420565 3127 ts_tablet_manager.cc:1918] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:21.420933 3127 log.cc:1196] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/26103012be924acb996660e417bd78f5
I20260619 18:58:21.421705 3127 ts_tablet_manager.cc:1939] T 26103012be924acb996660e417bd78f5 P 28333353307340039675050c246ac39e: Deleting consensus metadata
I20260619 18:58:21.424067 2611 catalog_manager.cc:4841] TS 28333353307340039675050c246ac39e (127.15.91.1:44223): tablet 26103012be924acb996660e417bd78f5 (table test-table [id=02dd33bc3f9f498fad4f4b72e323d6da]) successfully deleted
I20260619 18:58:21.425493 3125 ts_tablet_manager.cc:1918] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:21.425699 3125 log.cc:1196] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/26103012be924acb996660e417bd78f5
I20260619 18:58:21.426363 3125 ts_tablet_manager.cc:1939] T 26103012be924acb996660e417bd78f5 P 86f4b9a668c4423283f6363a32260883: Deleting consensus metadata
I20260619 18:58:21.428690 2609 catalog_manager.cc:4841] TS 86f4b9a668c4423283f6363a32260883 (127.15.91.2:39053): tablet 26103012be924acb996660e417bd78f5 (table test-table [id=02dd33bc3f9f498fad4f4b72e323d6da]) successfully deleted
I20260619 18:58:21.430536 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:21.378651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51836 (local address 127.15.91.1:44223)
0619 18:58:21.378975 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:21.378985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:21.403474 (+ 24489us) server_negotiation.cc:407] Connection header received
0619 18:58:21.407266 (+ 3792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:21.407273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:21.407377 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:21.407499 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:21.409539 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.411018 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.413138 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.413962 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.414095 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.420742 (+ 6647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:21.420768 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:21.420789 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:21.420853 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:21.425834 (+ 4981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:21.427496 (+ 1662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:21.427507 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:21.427511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:21.427761 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:21.428364 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:21.428370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:21.428373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:21.428976 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:21.429373 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:21.429770 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 18:58:21.430068 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260619 18:58:21.431680 2809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:21.560417 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 2685
I20260619 18:58:21.580291 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 2820
Jun 19 18:58:21 dist-test-slave-dm3f krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.2@KRBTEST.COM
I20260619 18:58:21.600805 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:21.581167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.15.91.2:39053)
0619 18:58:21.581348 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:21.581353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:21.581419 (+ 66us) server_negotiation.cc:407] Connection header received
0619 18:58:21.581566 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:21.581570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:21.581655 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:21.581764 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:21.583127 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.583881 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.585258 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.585686 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.585798 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.594482 (+ 8684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:21.594500 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:21.594518 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:21.594564 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:21.597875 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:21.598788 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:21.598804 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:21.598807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:21.598967 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:21.599352 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:21.599356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:21.599357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:21.599773 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:21.600013 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:21.600327 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 18:58:21.600523 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:58:21.601568 2943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:21.696453 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 2820
I20260619 18:58:21.715735 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 2954
Jun 19 18:58:21 dist-test-slave-dm3f krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.3@KRBTEST.COM
I20260619 18:58:21.735535 3098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:21.716923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.15.91.3:41811)
0619 18:58:21.717117 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:21.717122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:21.717143 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:58:21.717284 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:21.717288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:21.717372 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:21.717472 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:21.719744 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.720524 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.722109 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:21.722562 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:21.722690 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:21.728611 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:21.728631 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:21.728649 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:21.728698 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:21.731950 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:21.733234 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:21.733244 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:21.733247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:21.733455 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:21.733898 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:21.733902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:21.733904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:21.734311 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:21.734562 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:21.735030 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 18:58:21.735256 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:58:21.736295 3077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:21.828930 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 2954
I20260619 18:58:21.846866 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 2592
I20260619 18:58:21.848120 2654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:21.966846 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 2592
2026-06-19T18:58:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4840 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T18:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:22Z Disabled control of system clock
I20260619 18:58:22.051416 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:43537
--webserver_interface=127.15.91.62
--webserver_port=0
--builtin_ntp_servers=127.15.91.20:33651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:43537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:58:22.420006 3136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:22.420357 3136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:22.420444 3136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:22.428977 3136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:22.429122 3136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:22.429162 3136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:22.429214 3136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:22.429260 3136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:22.440142 3136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:43537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:43537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:22.441746 3136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:22.443796 3136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:22.458490 3141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:22.458437 3142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:22.458490 3144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:22.459359 3136 server_base.cc:1034] running on GCE node
I20260619 18:58:22.460520 3136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:22.462584 3136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:22.466192 3136 hybrid_clock.cc:648] HybridClock initialized: now 1781895502466091 us; error 80 us; skew 500 ppm
I20260619 18:58:22.466828 3136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:22.469935 3136 webserver.cc:460] Webserver started at http://127.15.91.62:34155/ using document root <none> and password file <none>
I20260619 18:58:22.470911 3136 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:22.471086 3136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:22.471477 3136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:22.475004 3136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "c2cc305c6e4849ffa2ffb509d8deb980"
format_stamp: "Formatted at 2026-06-19 18:58:22 on dist-test-slave-dm3f"
server_key: "2cf954d2bfe438086076ae76fceb9cc6"
server_key_iv: "0fd327e7004ae49b69853ee933a0a483"
server_key_version: "encryptionkey@0"
I20260619 18:58:22.476649 3136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "c2cc305c6e4849ffa2ffb509d8deb980"
format_stamp: "Formatted at 2026-06-19 18:58:22 on dist-test-slave-dm3f"
server_key: "2cf954d2bfe438086076ae76fceb9cc6"
server_key_iv: "0fd327e7004ae49b69853ee933a0a483"
server_key_version: "encryptionkey@0"
I20260619 18:58:22.483368 3136 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260619 18:58:22.487867 3150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:22.489626 3136 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:58:22.489893 3136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "c2cc305c6e4849ffa2ffb509d8deb980"
format_stamp: "Formatted at 2026-06-19 18:58:22 on dist-test-slave-dm3f"
server_key: "2cf954d2bfe438086076ae76fceb9cc6"
server_key_iv: "0fd327e7004ae49b69853ee933a0a483"
server_key_version: "encryptionkey@0"
I20260619 18:58:22.490188 3136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:22.513105 3136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:22.514155 3136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:22.514503 3136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:22.538525 3136 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:43537
I20260619 18:58:22.538540 3201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:43537 every 8 connection(s)
I20260619 18:58:22.540217 3136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:22.544806 3202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:22.550352 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 3136
I20260619 18:58:22.550565 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:22.551163 15724 external_mini_cluster.cc:1428] Setting key 06d37ef895ce12224a5c845cd6c1b6ec
I20260619 18:58:22.558007 3202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980: Bootstrap starting.
I20260619 18:58:22.562638 3202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:22.564198 3202 log.cc:824] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:22.568233 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:22.553450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.15.91.62:43537)
0619 18:58:22.554820 (+ 1370us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:22.554835 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:22.554890 (+ 55us) server_negotiation.cc:407] Connection header received
0619 18:58:22.556866 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:22.556896 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:22.557522 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:22.557929 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:22.558699 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:22.560394 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:22.562440 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:22.563384 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:22.563563 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:22.565125 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:22.565143 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:22.565168 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:22.566082 (+ 914us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:22.566187 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:22.566214 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:22.566326 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:22.566656 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 18:58:22.567136 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":477,"threads_started":1}
I20260619 18:58:22.568292 3202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980: No bootstrap required, opened a new log
I20260619 18:58:22.576543 3202 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cc305c6e4849ffa2ffb509d8deb980" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43537 } }
I20260619 18:58:22.577061 3202 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:22.577190 3202 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cc305c6e4849ffa2ffb509d8deb980, State: Initialized, Role: FOLLOWER
I20260619 18:58:22.577944 3202 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cc305c6e4849ffa2ffb509d8deb980" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43537 } }
I20260619 18:58:22.578261 3202 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:22.578426 3202 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:22.578656 3202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:22.581135 3202 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cc305c6e4849ffa2ffb509d8deb980" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43537 } }
I20260619 18:58:22.581845 3202 leader_election.cc:302] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2cc305c6e4849ffa2ffb509d8deb980; no voters:
I20260619 18:58:22.582549 3202 leader_election.cc:288] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:22.582880 3207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:22.584594 3207 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [term 1 LEADER]: Becoming Leader. State: Replica: c2cc305c6e4849ffa2ffb509d8deb980, State: Running, Role: LEADER
I20260619 18:58:22.585259 3202 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:22.585255 3207 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cc305c6e4849ffa2ffb509d8deb980" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43537 } }
I20260619 18:58:22.588560 3209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2cc305c6e4849ffa2ffb509d8deb980. Latest consensus state: current_term: 1 leader_uuid: "c2cc305c6e4849ffa2ffb509d8deb980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cc305c6e4849ffa2ffb509d8deb980" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43537 } } }
I20260619 18:58:22.589026 3208 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2cc305c6e4849ffa2ffb509d8deb980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cc305c6e4849ffa2ffb509d8deb980" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 43537 } } }
I20260619 18:58:22.589282 3209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:22.589329 3208 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:22.592442 3215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:22.599356 3215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:22.609485 3215 catalog_manager.cc:1347] Generated new cluster ID: 6d0884a481244953a656734d8fa36a0c
I20260619 18:58:22.609645 3215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:22.637679 3215 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:22.639250 3215 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:22.651955 3215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980: Generated new TSK 0
I20260619 18:58:22.652963 3215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:22.662179 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.1:0
--local_ip_for_outbound_sockets=127.15.91.1
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43537
--builtin_ntp_servers=127.15.91.20:33651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:58:23.039628 3226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:23.039964 3226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:23.040135 3226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:23.048413 3226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:23.048544 3226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:23.048733 3226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.1
I20260619 18:58:23.059142 3226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.91.1
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.1
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:23.060580 3226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:23.062778 3226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:23.077957 3231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:23.077826 3234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:23.077872 3232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:23.079078 3226 server_base.cc:1034] running on GCE node
I20260619 18:58:23.079865 3226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:23.081014 3226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:23.082295 3226 hybrid_clock.cc:648] HybridClock initialized: now 1781895503082208 us; error 75 us; skew 500 ppm
I20260619 18:58:23.082762 3226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:23.085691 3226 webserver.cc:460] Webserver started at http://127.15.91.1:41699/ using document root <none> and password file <none>
I20260619 18:58:23.086632 3226 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:23.086850 3226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:23.087230 3226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:23.090904 3226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/data/instance:
uuid: "ccaa8dd566244ba0968188bbc1330eea"
format_stamp: "Formatted at 2026-06-19 18:58:23 on dist-test-slave-dm3f"
server_key: "b71764c5613aa4021029582b31a90e4a"
server_key_iv: "e211ca3b6b2ca7ab6b07e3af5e562ba5"
server_key_version: "encryptionkey@0"
I20260619 18:58:23.091933 3226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance:
uuid: "ccaa8dd566244ba0968188bbc1330eea"
format_stamp: "Formatted at 2026-06-19 18:58:23 on dist-test-slave-dm3f"
server_key: "b71764c5613aa4021029582b31a90e4a"
server_key_iv: "e211ca3b6b2ca7ab6b07e3af5e562ba5"
server_key_version: "encryptionkey@0"
I20260619 18:58:23.098310 3226 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:58:23.102788 3240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:23.104616 3226 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:23.104852 3226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/wal
uuid: "ccaa8dd566244ba0968188bbc1330eea"
format_stamp: "Formatted at 2026-06-19 18:58:23 on dist-test-slave-dm3f"
server_key: "b71764c5613aa4021029582b31a90e4a"
server_key_iv: "e211ca3b6b2ca7ab6b07e3af5e562ba5"
server_key_version: "encryptionkey@0"
I20260619 18:58:23.105120 3226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:23.139093 3226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:23.140087 3226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:23.140419 3226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:23.141676 3226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:23.143590 3226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:23.143707 3226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:23.143863 3226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:23.143962 3226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:23.181895 3226 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.1:41141
I20260619 18:58:23.182013 3352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.1:41141 every 8 connection(s)
I20260619 18:58:23.183454 3226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/data/info.pb
I20260619 18:58:23.183926 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 3226
I20260619 18:58:23.184144 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/wal/instance
I20260619 18:58:23.184813 15724 external_mini_cluster.cc:1428] Setting key 9d3d4eef4b108e283a0372011b832460
I20260619 18:58:23.188223 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.2:0
--local_ip_for_outbound_sockets=127.15.91.2
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43537
--builtin_ntp_servers=127.15.91.20:33651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:58:23.202450 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:23.186876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:57265 (local address 127.15.91.62:43537)
0619 18:58:23.187401 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:23.187407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:23.189943 (+ 2536us) server_negotiation.cc:407] Connection header received
0619 18:58:23.193139 (+ 3196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:23.193144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:23.193245 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:23.193410 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:23.195284 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:23.196154 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:23.198569 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:23.199400 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:23.199589 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:23.201340 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:23.201351 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:23.201359 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:23.201390 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:23.201448 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:23.201468 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:23.201579 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:23.201830 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 18:58:23.201964 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":257,"threads_started":1}
I20260619 18:58:23.203439 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:23.187733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43537 (local address 127.15.91.1:57265)
0619 18:58:23.189623 (+ 1890us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:23.189714 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:23.192674 (+ 2960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:23.193754 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:23.193770 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:23.193920 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:23.194970 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:23.195006 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:23.196397 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:23.196407 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:23.198327 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:23.198348 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:23.199677 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:23.199682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:23.199929 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:23.200883 (+ 954us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:58:23.200886 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:23.200935 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:58:23.201044 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:23.201591 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:23.201600 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:23.201606 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:58:23.201774 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 18:58:23.202153 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":669,"threads_started":1}
I20260619 18:58:23.205358 3353 heartbeater.cc:347] Connected to a master server at 127.15.91.62:43537
I20260619 18:58:23.205725 3353 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:23.206558 3353 heartbeater.cc:511] Master 127.15.91.62:43537 requested a full tablet report, sending...
I20260619 18:58:23.209143 3167 ts_manager.cc:194] Registered new tserver with Master: ccaa8dd566244ba0968188bbc1330eea (127.15.91.1:41141)
I20260619 18:58:23.211058 3167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.1:57265
W20260619 18:58:23.569944 3357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:23.570269 3357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:23.570422 3357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:23.579054 3357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:23.579181 3357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:23.579375 3357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.2
I20260619 18:58:23.589493 3357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.91.2
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.2
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:23.591061 3357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:23.593098 3357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:23.608940 3362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:23.609097 3365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:23.608928 3363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:23.610042 3357 server_base.cc:1034] running on GCE node
I20260619 18:58:23.610886 3357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:23.612087 3357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:23.613382 3357 hybrid_clock.cc:648] HybridClock initialized: now 1781895503613333 us; error 69 us; skew 500 ppm
I20260619 18:58:23.613808 3357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:23.616792 3357 webserver.cc:460] Webserver started at http://127.15.91.2:46211/ using document root <none> and password file <none>
I20260619 18:58:23.617710 3357 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:23.617889 3357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:23.618263 3357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:23.621740 3357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/data/instance:
uuid: "865bb8c1974a4b3b925534414ceb0432"
format_stamp: "Formatted at 2026-06-19 18:58:23 on dist-test-slave-dm3f"
server_key: "45350ae52bca27bc7f5df152f29cc278"
server_key_iv: "6f1aaa5be46c337df381c915de92bb04"
server_key_version: "encryptionkey@0"
I20260619 18:58:23.622877 3357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance:
uuid: "865bb8c1974a4b3b925534414ceb0432"
format_stamp: "Formatted at 2026-06-19 18:58:23 on dist-test-slave-dm3f"
server_key: "45350ae52bca27bc7f5df152f29cc278"
server_key_iv: "6f1aaa5be46c337df381c915de92bb04"
server_key_version: "encryptionkey@0"
I20260619 18:58:23.629209 3357 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:58:23.633747 3371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:23.635597 3357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:58:23.635927 3357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/wal
uuid: "865bb8c1974a4b3b925534414ceb0432"
format_stamp: "Formatted at 2026-06-19 18:58:23 on dist-test-slave-dm3f"
server_key: "45350ae52bca27bc7f5df152f29cc278"
server_key_iv: "6f1aaa5be46c337df381c915de92bb04"
server_key_version: "encryptionkey@0"
I20260619 18:58:23.636232 3357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:23.666360 3357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:23.667419 3357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:23.667760 3357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:23.668937 3357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:23.670820 3357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:23.670982 3357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:23.671128 3357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:23.671228 3357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:23.711486 3357 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.2:40389
I20260619 18:58:23.711556 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.2:40389 every 8 connection(s)
I20260619 18:58:23.713075 3357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/data/info.pb
I20260619 18:58:23.721597 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 3357
I20260619 18:58:23.721818 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/wal/instance
I20260619 18:58:23.722378 15724 external_mini_cluster.cc:1428] Setting key 6f1f20cf01e00d965577db78d8b6e852
I20260619 18:58:23.726293 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.91.3:0
--local_ip_for_outbound_sockets=127.15.91.3
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43537
--builtin_ntp_servers=127.15.91.20:33651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:58:23.730918 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:23.716239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.2:47621 (local address 127.15.91.62:43537)
0619 18:58:23.716814 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:23.716820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:23.718557 (+ 1737us) server_negotiation.cc:407] Connection header received
0619 18:58:23.720764 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:23.720770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:23.720860 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:23.720979 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:23.722850 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:23.723879 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:23.726558 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:23.727164 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:23.727306 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:23.729605 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:23.729612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:23.729615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:23.729647 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:23.729698 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:23.729713 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:23.729815 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:23.730244 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 18:58:23.730425 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":303,"threads_started":1}
I20260619 18:58:23.732460 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:23.716980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43537 (local address 127.15.91.2:47621)
0619 18:58:23.718297 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:23.718354 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:23.720330 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:23.721340 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:23.721357 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:23.721512 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:23.722507 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:23.722543 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:23.724108 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:23.724114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:23.726264 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:23.726293 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:23.727486 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:23.727499 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:23.727919 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:23.728887 (+ 968us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:58:23.728893 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:23.728970 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:58:23.729102 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:23.729867 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:23.729882 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:23.729895 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 18:58:23.730148 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 18:58:23.730699 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":472,"threads_started":1}
I20260619 18:58:23.734871 3485 heartbeater.cc:347] Connected to a master server at 127.15.91.62:43537
I20260619 18:58:23.735294 3485 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:23.736266 3485 heartbeater.cc:511] Master 127.15.91.62:43537 requested a full tablet report, sending...
I20260619 18:58:23.738348 3167 ts_manager.cc:194] Registered new tserver with Master: 865bb8c1974a4b3b925534414ceb0432 (127.15.91.2:40389)
I20260619 18:58:23.739332 3167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.2:47621
W20260619 18:58:24.101361 3489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:24.101774 3489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:24.101946 3489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:24.109936 3489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:24.110106 3489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:24.110301 3489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.91.3
I20260619 18:58:24.120560 3489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.91.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.91.3
--webserver_port=0
--tserver_master_addrs=127.15.91.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.91.3
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:24.122068 3489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:24.124168 3489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:24.140547 3495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:24.140852 3494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:24.141249 3497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:24.141605 3489 server_base.cc:1034] running on GCE node
I20260619 18:58:24.142329 3489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:24.143612 3489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:24.144905 3489 hybrid_clock.cc:648] HybridClock initialized: now 1781895504144851 us; error 61 us; skew 500 ppm
I20260619 18:58:24.145450 3489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:58:24.148701 3489 webserver.cc:460] Webserver started at http://127.15.91.3:43275/ using document root <none> and password file <none>
I20260619 18:58:24.149596 3489 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:24.149771 3489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:24.150146 3489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:24.153573 3489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/data/instance:
uuid: "2b6ea8d21600478998009c2d99b6252a"
format_stamp: "Formatted at 2026-06-19 18:58:24 on dist-test-slave-dm3f"
server_key: "88b81259e993027c1714c7bd6af70344"
server_key_iv: "741ea4a819a1cbd1540fa76015b8aa39"
server_key_version: "encryptionkey@0"
I20260619 18:58:24.154575 3489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance:
uuid: "2b6ea8d21600478998009c2d99b6252a"
format_stamp: "Formatted at 2026-06-19 18:58:24 on dist-test-slave-dm3f"
server_key: "88b81259e993027c1714c7bd6af70344"
server_key_iv: "741ea4a819a1cbd1540fa76015b8aa39"
server_key_version: "encryptionkey@0"
I20260619 18:58:24.161679 3489 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 18:58:24.167124 3503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:24.168848 3489 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:24.169112 3489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/wal
uuid: "2b6ea8d21600478998009c2d99b6252a"
format_stamp: "Formatted at 2026-06-19 18:58:24 on dist-test-slave-dm3f"
server_key: "88b81259e993027c1714c7bd6af70344"
server_key_iv: "741ea4a819a1cbd1540fa76015b8aa39"
server_key_version: "encryptionkey@0"
I20260619 18:58:24.169396 3489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:24.188009 3489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:24.189033 3489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:24.189390 3489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:24.190778 3489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:58:24.192752 3489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:58:24.192924 3489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:24.193076 3489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:58:24.193184 3489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:24.216110 3353 heartbeater.cc:503] Master 127.15.91.62:43537 was elected leader, sending a full tablet report...
I20260619 18:58:24.233496 3489 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.3:45745
I20260619 18:58:24.233543 3615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.3:45745 every 8 connection(s)
I20260619 18:58:24.235146 3489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/data/info.pb
I20260619 18:58:24.239252 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 3489
I20260619 18:58:24.239449 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/wal/instance
I20260619 18:58:24.240077 15724 external_mini_cluster.cc:1428] Setting key a2923873c3b928563d3eed9740dd296e
I20260619 18:58:24.251346 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.238356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.3:41905 (local address 127.15.91.62:43537)
0619 18:58:24.238943 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:24.238950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:24.240947 (+ 1997us) server_negotiation.cc:407] Connection header received
0619 18:58:24.243581 (+ 2634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:24.243588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:24.243687 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:24.243818 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:24.245166 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.246022 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.248118 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.248571 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.248701 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.250382 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:24.250389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:24.250392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:24.250415 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:24.250463 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:24.250476 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:24.250566 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:24.250904 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 18:58:24.251008 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":270,"threads_started":1}
I20260619 18:58:24.252439 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.239166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:43537 (local address 127.15.91.3:41905)
0619 18:58:24.240722 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.240771 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:24.243189 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:24.244120 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:24.244130 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:24.244245 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:58:24.244941 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.244961 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.246271 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.246276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.247864 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.247883 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.248891 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.248897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.249150 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.249954 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:58:24.249957 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:58:24.250013 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:58:24.250111 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:58:24.250586 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:58:24.250594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:58:24.250601 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:58:24.250871 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 18:58:24.251251 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":397,"threads_started":1}
I20260619 18:58:24.254199 3616 heartbeater.cc:347] Connected to a master server at 127.15.91.62:43537
I20260619 18:58:24.254575 3616 heartbeater.cc:464] Registering TS with master...
I20260619 18:58:24.255647 3616 heartbeater.cc:511] Master 127.15.91.62:43537 requested a full tablet report, sending...
I20260619 18:58:24.257234 3167 ts_manager.cc:194] Registered new tserver with Master: 2b6ea8d21600478998009c2d99b6252a (127.15.91.3:45745)
I20260619 18:58:24.258116 3167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.91.3:41905
I20260619 18:58:24.270182 15724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:58:24.282456 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.275096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.15.91.62:43537)
0619 18:58:24.275357 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:24.275364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:24.275927 (+ 563us) server_negotiation.cc:407] Connection header received
0619 18:58:24.276056 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:24.276061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:24.276193 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:24.276445 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:24.277180 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.277962 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.279844 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.280327 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.280468 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.281698 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:24.281704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:24.281707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:24.281749 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:24.281786 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:24.281800 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:24.281885 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:24.282073 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 18:58:24.282172 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 18:58:24.287683 3167 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:58:24.295374 3167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:58:24.332377 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.310508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.15.91.1:41141)
0619 18:58:24.322530 (+ 12022us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:24.322542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:24.322582 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:58:24.322751 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:24.322761 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:24.323077 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:24.323326 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:24.324586 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.326277 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.329419 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.330942 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.331106 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.331451 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:24.331785 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 18:58:24.331989 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11800,"thread_start_us":11614,"threads_started":1}
I20260619 18:58:24.332362 3630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.310226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:41141 (local address 127.0.0.1:53856)
0619 18:58:24.321280 (+ 11054us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.321363 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:24.321704 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:24.323677 (+ 1973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:24.323687 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:24.323738 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:24.324318 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.324335 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.326473 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.326479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.329122 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.329138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.331196 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.331202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.331472 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.331534 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 18:58:24.331704 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 18:58:24.331959 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10546,"mutex_wait_us":4,"thread_start_us":12826,"threads_started":1}
I20260619 18:58:24.333963 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.310556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:45745 (local address 127.0.0.1:43326)
0619 18:58:24.322821 (+ 12265us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.322842 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:24.322984 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:24.325258 (+ 2274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:24.325263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:24.325294 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:24.325771 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.325790 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.328012 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.328018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.330305 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.330323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.333095 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.333104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.333340 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.333393 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 18:58:24.333470 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:58:24.333638 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12101,"thread_start_us":9967,"threads_started":1}
I20260619 18:58:24.335391 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.311292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.15.91.3:45745)
0619 18:58:24.323346 (+ 12054us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:24.323353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:24.323388 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:58:24.323519 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:24.323527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:24.324692 (+ 1165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:24.324945 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:24.326077 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.327837 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.330564 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.332819 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.334534 (+ 1715us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.334846 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:24.334920 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 18:58:24.335070 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11867,"thread_start_us":11684,"threads_started":1}
I20260619 18:58:24.336546 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.323572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:40389 (local address 127.0.0.1:39418)
0619 18:58:24.324942 (+ 1370us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.324971 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:24.325148 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:24.327119 (+ 1971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:24.327125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:24.327157 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:24.327609 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.327621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.330187 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.330193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.333493 (+ 3300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.333514 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.335610 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.335616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.335836 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.335879 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 18:58:24.335990 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 18:58:24.336155 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":30,"thread_start_us":351,"threads_started":1}
I20260619 18:58:24.336618 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.323834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.15.91.2:40389)
0619 18:58:24.326420 (+ 2586us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:24.326431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:24.326472 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:58:24.326564 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:24.326570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:24.326891 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:24.327129 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:24.328156 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.329966 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.333771 (+ 3805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.335333 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.335509 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.335806 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:24.336080 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 18:58:24.336292 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2374,"thread_start_us":2256,"threads_started":1}
I20260619 18:58:24.338045 3288 tablet_service.cc:1467] Processing CreateTablet for tablet 346966c4fc5d432d9d6e4ba61612618f (DEFAULT_TABLE table=test-table [id=1bd4e4e653134bc2b3ff4d64f9a80a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:24.340021 3288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 346966c4fc5d432d9d6e4ba61612618f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:24.340253 3551 tablet_service.cc:1467] Processing CreateTablet for tablet 346966c4fc5d432d9d6e4ba61612618f (DEFAULT_TABLE table=test-table [id=1bd4e4e653134bc2b3ff4d64f9a80a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:24.341954 3551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 346966c4fc5d432d9d6e4ba61612618f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:24.341573 3419 tablet_service.cc:1467] Processing CreateTablet for tablet 346966c4fc5d432d9d6e4ba61612618f (DEFAULT_TABLE table=test-table [id=1bd4e4e653134bc2b3ff4d64f9a80a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:58:24.343300 3419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 346966c4fc5d432d9d6e4ba61612618f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:24.358886 3635 tablet_bootstrap.cc:492] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: Bootstrap starting.
I20260619 18:58:24.359915 3636 tablet_bootstrap.cc:492] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: Bootstrap starting.
I20260619 18:58:24.363619 3637 tablet_bootstrap.cc:492] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: Bootstrap starting.
I20260619 18:58:24.363749 3635 tablet_bootstrap.cc:654] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:24.363843 3636 tablet_bootstrap.cc:654] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:24.365353 3636 log.cc:824] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:24.365353 3635 log.cc:824] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:24.368287 3637 tablet_bootstrap.cc:654] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:24.369524 3637 log.cc:824] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:24.373507 3637 tablet_bootstrap.cc:492] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: No bootstrap required, opened a new log
I20260619 18:58:24.373526 3636 tablet_bootstrap.cc:492] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: No bootstrap required, opened a new log
I20260619 18:58:24.373601 3635 tablet_bootstrap.cc:492] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: No bootstrap required, opened a new log
I20260619 18:58:24.373919 3637 ts_tablet_manager.cc:1397] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 18:58:24.373911 3636 ts_tablet_manager.cc:1397] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20260619 18:58:24.373910 3635 ts_tablet_manager.cc:1397] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260619 18:58:24.383862 3637 raft_consensus.cc:348] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa8dd566244ba0968188bbc1330eea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41141 } } peers { permanent_uuid: "2b6ea8d21600478998009c2d99b6252a" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45745 } } peers { permanent_uuid: "865bb8c1974a4b3b925534414ceb0432" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40389 } }
I20260619 18:58:24.384397 3637 raft_consensus.cc:374] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:24.384562 3637 raft_consensus.cc:729] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865bb8c1974a4b3b925534414ceb0432, State: Initialized, Role: FOLLOWER
I20260619 18:58:24.385401 3637 consensus_queue.cc:260] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa8dd566244ba0968188bbc1330eea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41141 } } peers { permanent_uuid: "2b6ea8d21600478998009c2d99b6252a" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45745 } } peers { permanent_uuid: "865bb8c1974a4b3b925534414ceb0432" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40389 } }
I20260619 18:58:24.386324 3635 raft_consensus.cc:348] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa8dd566244ba0968188bbc1330eea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41141 } } peers { permanent_uuid: "2b6ea8d21600478998009c2d99b6252a" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45745 } } peers { permanent_uuid: "865bb8c1974a4b3b925534414ceb0432" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40389 } }
I20260619 18:58:24.386858 3485 heartbeater.cc:503] Master 127.15.91.62:43537 was elected leader, sending a full tablet report...
I20260619 18:58:24.386987 3635 raft_consensus.cc:374] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:24.387095 3635 raft_consensus.cc:729] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6ea8d21600478998009c2d99b6252a, State: Initialized, Role: FOLLOWER
I20260619 18:58:24.386804 3636 raft_consensus.cc:348] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa8dd566244ba0968188bbc1330eea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41141 } } peers { permanent_uuid: "2b6ea8d21600478998009c2d99b6252a" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45745 } } peers { permanent_uuid: "865bb8c1974a4b3b925534414ceb0432" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40389 } }
I20260619 18:58:24.387265 3636 raft_consensus.cc:374] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:24.387353 3636 raft_consensus.cc:729] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccaa8dd566244ba0968188bbc1330eea, State: Initialized, Role: FOLLOWER
I20260619 18:58:24.387655 3637 ts_tablet_manager.cc:1428] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260619 18:58:24.387972 3636 consensus_queue.cc:260] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa8dd566244ba0968188bbc1330eea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41141 } } peers { permanent_uuid: "2b6ea8d21600478998009c2d99b6252a" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45745 } } peers { permanent_uuid: "865bb8c1974a4b3b925534414ceb0432" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40389 } }
I20260619 18:58:24.387972 3635 consensus_queue.cc:260] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa8dd566244ba0968188bbc1330eea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41141 } } peers { permanent_uuid: "2b6ea8d21600478998009c2d99b6252a" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45745 } } peers { permanent_uuid: "865bb8c1974a4b3b925534414ceb0432" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40389 } }
I20260619 18:58:24.389828 3616 heartbeater.cc:503] Master 127.15.91.62:43537 was elected leader, sending a full tablet report...
I20260619 18:58:24.390329 3636 ts_tablet_manager.cc:1428] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260619 18:58:24.390455 3635 ts_tablet_manager.cc:1428] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: Time spent starting tablet: real 0.016s user 0.017s sys 0.001s
W20260619 18:58:24.436486 3354 tablet.cc:2369] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:58:24.465703 3486 tablet.cc:2369] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:58:24.480086 3643 raft_consensus.cc:482] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:58:24.480428 3643 raft_consensus.cc:504] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa8dd566244ba0968188bbc1330eea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41141 } } peers { permanent_uuid: "2b6ea8d21600478998009c2d99b6252a" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45745 } } peers { permanent_uuid: "865bb8c1974a4b3b925534414ceb0432" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40389 } }
I20260619 18:58:24.482172 3643 leader_election.cc:288] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b6ea8d21600478998009c2d99b6252a (127.15.91.3:45745), 865bb8c1974a4b3b925534414ceb0432 (127.15.91.2:40389)
W20260619 18:58:24.487201 3617 tablet.cc:2369] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:58:24.490661 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.482936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:40951 (local address 127.15.91.2:40389)
0619 18:58:24.483182 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:24.483191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:24.484644 (+ 1453us) server_negotiation.cc:407] Connection header received
0619 18:58:24.484858 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:24.484863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:24.484947 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:24.485047 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:24.485643 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.486355 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.488670 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.489477 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.489652 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.489762 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:24.490210 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 18:58:24.490360 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:58:24.490736 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.482847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.91.1:41281 (local address 127.15.91.3:45745)
0619 18:58:24.483057 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:24.483064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:24.484156 (+ 1092us) server_negotiation.cc:407] Connection header received
0619 18:58:24.484405 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:24.484411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:24.484491 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:24.484597 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:58:24.485158 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.486005 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.488867 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.489706 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.489874 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.489997 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:24.490312 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 18:58:24.490451 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:58:24.490821 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.483425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.2:40389 (local address 127.15.91.1:40951)
0619 18:58:24.484520 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.484536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:24.484697 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:24.485078 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:24.485082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:24.485107 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:24.485495 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.485506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.486504 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.486511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.488391 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.488412 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.489721 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.489728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.490023 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.490054 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:58:24.490166 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 18:58:24.490330 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":109,"thread_start_us":741,"threads_started":1}
I20260619 18:58:24.490898 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.482776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.3:45745 (local address 127.15.91.1:41281)
0619 18:58:24.484021 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.484043 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:58:24.484245 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:58:24.484616 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:58:24.484620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:58:24.484650 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:58:24.485006 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.485015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.486151 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.486158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.488623 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:58:24.488640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.489936 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.489943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:58:24.490141 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.490167 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:58:24.490282 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 18:58:24.490425 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":105,"thread_start_us":641,"threads_started":1}
I20260619 18:58:24.492046 3571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "346966c4fc5d432d9d6e4ba61612618f" candidate_uuid: "ccaa8dd566244ba0968188bbc1330eea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b6ea8d21600478998009c2d99b6252a" is_pre_election: true
I20260619 18:58:24.492295 3439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "346966c4fc5d432d9d6e4ba61612618f" candidate_uuid: "ccaa8dd566244ba0968188bbc1330eea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865bb8c1974a4b3b925534414ceb0432" is_pre_election: true
I20260619 18:58:24.492606 3571 raft_consensus.cc:2454] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccaa8dd566244ba0968188bbc1330eea in term 0.
I20260619 18:58:24.492923 3439 raft_consensus.cc:2454] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccaa8dd566244ba0968188bbc1330eea in term 0.
I20260619 18:58:24.493526 3241 leader_election.cc:302] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b6ea8d21600478998009c2d99b6252a, ccaa8dd566244ba0968188bbc1330eea; no voters:
I20260619 18:58:24.493978 3643 raft_consensus.cc:2781] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:58:24.494199 3643 raft_consensus.cc:482] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:58:24.494308 3643 raft_consensus.cc:3037] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:24.496773 3643 raft_consensus.cc:504] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa8dd566244ba0968188bbc1330eea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41141 } } peers { permanent_uuid: "2b6ea8d21600478998009c2d99b6252a" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45745 } } peers { permanent_uuid: "865bb8c1974a4b3b925534414ceb0432" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40389 } }
I20260619 18:58:24.497538 3643 leader_election.cc:288] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [CANDIDATE]: Term 1 election: Requested vote from peers 2b6ea8d21600478998009c2d99b6252a (127.15.91.3:45745), 865bb8c1974a4b3b925534414ceb0432 (127.15.91.2:40389)
I20260619 18:58:24.498149 3571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "346966c4fc5d432d9d6e4ba61612618f" candidate_uuid: "ccaa8dd566244ba0968188bbc1330eea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b6ea8d21600478998009c2d99b6252a"
I20260619 18:58:24.498149 3439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "346966c4fc5d432d9d6e4ba61612618f" candidate_uuid: "ccaa8dd566244ba0968188bbc1330eea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865bb8c1974a4b3b925534414ceb0432"
I20260619 18:58:24.498406 3439 raft_consensus.cc:3037] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:24.498406 3571 raft_consensus.cc:3037] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:24.501987 3571 raft_consensus.cc:2454] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccaa8dd566244ba0968188bbc1330eea in term 1.
I20260619 18:58:24.501987 3439 raft_consensus.cc:2454] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccaa8dd566244ba0968188bbc1330eea in term 1.
I20260619 18:58:24.502658 3244 leader_election.cc:302] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b6ea8d21600478998009c2d99b6252a, 865bb8c1974a4b3b925534414ceb0432, ccaa8dd566244ba0968188bbc1330eea; no voters:
I20260619 18:58:24.503160 3643 raft_consensus.cc:2781] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:24.503685 3643 raft_consensus.cc:686] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 1 LEADER]: Becoming Leader. State: Replica: ccaa8dd566244ba0968188bbc1330eea, State: Running, Role: LEADER
I20260619 18:58:24.504256 3643 consensus_queue.cc:237] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa8dd566244ba0968188bbc1330eea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41141 } } peers { permanent_uuid: "2b6ea8d21600478998009c2d99b6252a" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45745 } } peers { permanent_uuid: "865bb8c1974a4b3b925534414ceb0432" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40389 } }
I20260619 18:58:24.509574 3167 catalog_manager.cc:5495] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea reported cstate change: term changed from 0 to 1, leader changed from <none> to ccaa8dd566244ba0968188bbc1330eea (127.15.91.1). New cstate: current_term: 1 leader_uuid: "ccaa8dd566244ba0968188bbc1330eea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa8dd566244ba0968188bbc1330eea" member_type: VOTER last_known_addr { host: "127.15.91.1" port: 41141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b6ea8d21600478998009c2d99b6252a" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "865bb8c1974a4b3b925534414ceb0432" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40389 } health_report { overall_health: UNKNOWN } } }
I20260619 18:58:24.534982 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.528798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.15.91.1:41141)
0619 18:58:24.529019 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:24.529026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:24.529115 (+ 89us) server_negotiation.cc:407] Connection header received
0619 18:58:24.529412 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:24.529416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:24.529522 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:24.529699 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:58:24.530102 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.530944 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.532522 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.532977 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.533126 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.533755 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:58:24.534381 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:58:24.534519 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:24.534612 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 18:58:24.534692 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:58:24.545204 3571 raft_consensus.cc:1261] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a [term 1 FOLLOWER]: Refusing update from remote peer ccaa8dd566244ba0968188bbc1330eea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:24.545277 3439 raft_consensus.cc:1261] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432 [term 1 FOLLOWER]: Refusing update from remote peer ccaa8dd566244ba0968188bbc1330eea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:58:24.546543 3646 consensus_queue.cc:1035] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [LEADER]: Connected to new peer: Peer: permanent_uuid: "865bb8c1974a4b3b925534414ceb0432" member_type: VOTER last_known_addr { host: "127.15.91.2" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:24.547166 3643 consensus_queue.cc:1035] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b6ea8d21600478998009c2d99b6252a" member_type: VOTER last_known_addr { host: "127.15.91.3" port: 45745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:58:24.580394 3166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:24.580705 3166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:58:24.585295 3166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2cc305c6e4849ffa2ffb509d8deb980: Sending DeleteTablet for 3 replicas of tablet 346966c4fc5d432d9d6e4ba61612618f
I20260619 18:58:24.586985 3551 tablet_service.cc:1514] Processing DeleteTablet for tablet 346966c4fc5d432d9d6e4ba61612618f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:24 UTC) from {username='slave'} at 127.0.0.1:43326
I20260619 18:58:24.587126 3288 tablet_service.cc:1514] Processing DeleteTablet for tablet 346966c4fc5d432d9d6e4ba61612618f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:24 UTC) from {username='slave'} at 127.0.0.1:53856
I20260619 18:58:24.588295 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 3226
I20260619 18:58:24.588263 3419 tablet_service.cc:1514] Processing DeleteTablet for tablet 346966c4fc5d432d9d6e4ba61612618f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:58:24 UTC) from {username='slave'} at 127.0.0.1:39418
I20260619 18:58:24.598148 3661 tablet_replica.cc:330] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: stopping tablet replica
I20260619 18:58:24.598953 3659 tablet_replica.cc:330] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: stopping tablet replica
I20260619 18:58:24.598923 3661 raft_consensus.cc:2229] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:58:24.599670 3661 raft_consensus.cc:2258] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:24.600806 3659 raft_consensus.cc:2229] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:58:24.601536 3659 raft_consensus.cc:2258] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:24.602516 3660 tablet_replica.cc:330] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: stopping tablet replica
I20260619 18:58:24.603101 3660 raft_consensus.cc:2229] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:58:24.603956 3660 raft_consensus.cc:2258] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:58:24.604648 3659 ts_tablet_manager.cc:1905] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:24.606551 3660 ts_tablet_manager.cc:1905] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:24.607466 3661 ts_tablet_manager.cc:1905] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:58:24.612982 3659 ts_tablet_manager.cc:1918] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:24.613221 3659 log.cc:1196] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-2/wal/wals/346966c4fc5d432d9d6e4ba61612618f
I20260619 18:58:24.613801 3659 ts_tablet_manager.cc:1939] T 346966c4fc5d432d9d6e4ba61612618f P 2b6ea8d21600478998009c2d99b6252a: Deleting consensus metadata
I20260619 18:58:24.616056 3151 catalog_manager.cc:4841] TS 2b6ea8d21600478998009c2d99b6252a (127.15.91.3:45745): tablet 346966c4fc5d432d9d6e4ba61612618f (table test-table [id=1bd4e4e653134bc2b3ff4d64f9a80a2c]) successfully deleted
I20260619 18:58:24.617332 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.589140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.15.91.1:41141)
0619 18:58:24.589388 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:24.589394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:24.597584 (+ 8190us) server_negotiation.cc:407] Connection header received
0619 18:58:24.599607 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:24.599620 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:24.599757 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:24.599859 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:24.602844 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.604233 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.609628 (+ 5395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.610450 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.610608 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.613944 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:24.613959 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:24.613979 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:24.615127 (+ 1148us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:24.615218 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:24.615268 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:24.615404 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:24.616724 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0619 18:58:24.616867 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:58:24.618625 3660 ts_tablet_manager.cc:1918] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:24.618825 3348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:24.619045 3660 log.cc:1196] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-0/wal/wals/346966c4fc5d432d9d6e4ba61612618f
I20260619 18:58:24.620502 3661 ts_tablet_manager.cc:1918] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:58:24.620808 3661 log.cc:1196] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: Deleting WAL directory at /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781895164554191-15724-0/minicluster-data/ts-1/wal/wals/346966c4fc5d432d9d6e4ba61612618f
I20260619 18:58:24.621577 3661 ts_tablet_manager.cc:1939] T 346966c4fc5d432d9d6e4ba61612618f P 865bb8c1974a4b3b925534414ceb0432: Deleting consensus metadata
I20260619 18:58:24.624207 3154 catalog_manager.cc:4841] TS 865bb8c1974a4b3b925534414ceb0432 (127.15.91.2:40389): tablet 346966c4fc5d432d9d6e4ba61612618f (table test-table [id=1bd4e4e653134bc2b3ff4d64f9a80a2c]) successfully deleted
I20260619 18:58:24.722961 3660 ts_tablet_manager.cc:1939] T 346966c4fc5d432d9d6e4ba61612618f P ccaa8dd566244ba0968188bbc1330eea: Deleting consensus metadata
I20260619 18:58:24.723767 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 3226
I20260619 18:58:24.745072 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 3357
W20260619 18:58:24.745242 3154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.91.1:41141 (error 108)
I20260619 18:58:24.747342 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.746890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:41141 (local address 127.0.0.1:49556)
0619 18:58:24.747064 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.747189 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 18:58:24.747624 3154 catalog_manager.cc:4584] TS ccaa8dd566244ba0968188bbc1330eea (127.15.91.1:41141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 346966c4fc5d432d9d6e4ba61612618f: Network error: Client connection negotiation failed: client connection to 127.15.91.1:41141: connect: Connection refused (error 111)
I20260619 18:58:24.754235 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.745998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.15.91.2:40389)
0619 18:58:24.746211 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:24.746221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:24.746247 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:58:24.746438 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:24.746443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:24.746547 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:24.746682 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:24.747143 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.748091 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.749545 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.749999 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.750158 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.751610 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:24.751620 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:24.751636 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:24.752855 (+ 1219us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:24.753036 (+ 181us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:24.753094 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:24.753352 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:24.753601 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 18:58:24.753776 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:58:24.755321 3480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:24.801789 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.801121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:41141 (local address 127.0.0.1:49568)
0619 18:58:24.801414 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.801537 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260619 18:58:24.803207 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.802680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:41141 (local address 127.0.0.1:49578)
0619 18:58:24.802922 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.803023 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 18:58:24.850195 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 3357
I20260619 18:58:24.864037 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.863458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:41141 (local address 127.0.0.1:49580)
0619 18:58:24.863717 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.863840 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 18:58:24.865417 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.864960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:41141 (local address 127.0.0.1:49592)
0619 18:58:24.865152 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.865248 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 18:58:24.869133 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 3489
I20260619 18:58:24.877472 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.870097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.15.91.3:45745)
0619 18:58:24.870507 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:24.870512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:24.870534 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:58:24.870599 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:24.870604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:24.870696 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:24.870903 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:24.871400 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.872219 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.873652 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:24.874082 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:24.874197 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:24.875796 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:24.875806 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:24.875822 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:24.876476 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:24.876533 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:24.876559 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:24.876673 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:24.876864 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 18:58:24.876991 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:58:24.878252 3611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:24.946790 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.946108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:41141 (local address 127.0.0.1:49596)
0619 18:58:24.946376 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.946498 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 18:58:24.948128 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:24.947606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.1:41141 (local address 127.0.0.1:49612)
0619 18:58:24.947854 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 18:58:24.947971 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260619 18:58:24.971881 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 3489
I20260619 18:58:24.989342 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 3136
I20260619 18:58:24.990638 3197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:25.103576 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 3136
2026-06-19T18:58:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3138 ms)
[----------] 6 tests from EncryptionPolicyTest (24045 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:25 dist-test-slave-dm3f krb5kdc[3670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:25 dist-test-slave-dm3f krb5kdc[3670](info): set up 2 sockets
Jun 19 18:58:25 dist-test-slave-dm3f krb5kdc[3670](info): commencing operation
krb5kdc: starting...
W20260619 18:58:26.239449 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:26 dist-test-slave-dm3f krb5kdc[3670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:26.522959 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:26.893337 3683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:26.893682 3683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:26.893850 3683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:26.901844 3683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:26.901998 3683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:26.902093 3683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:26.902215 3683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:26.902308 3683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:26.913157 3683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:26.914678 3683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:26.916263 3683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:26.931806 3689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:26.931840 3688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:26.934641 3691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:26.934890 3683 server_base.cc:1034] running on GCE node
I20260619 18:58:26.935978 3683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:26.937834 3683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:26.939106 3683 hybrid_clock.cc:648] HybridClock initialized: now 1781895506939021 us; error 93 us; skew 500 ppm
I20260619 18:58:26.939929 3683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:26 dist-test-slave-dm3f krb5kdc[3670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:26.947551 3683 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:26.950003 3683 webserver.cc:460] Webserver started at http://127.0.0.1:39685/ using document root <none> and password file <none>
I20260619 18:58:26.951177 3683 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:26.951381 3683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:26.951839 3683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:26.955803 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "a65d928d6ec44f7aa74ef53e2904b890"
format_stamp: "Formatted at 2026-06-19 18:58:26 on dist-test-slave-dm3f"
server_key: "a2ec4ae587a580fe22dcd6fed04cff42"
server_key_iv: "ebf045fa0c1ffe49f6d02f9f01023698"
server_key_version: "encryptionkey@0"
I20260619 18:58:26.956842 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "a65d928d6ec44f7aa74ef53e2904b890"
format_stamp: "Formatted at 2026-06-19 18:58:26 on dist-test-slave-dm3f"
server_key: "a2ec4ae587a580fe22dcd6fed04cff42"
server_key_iv: "ebf045fa0c1ffe49f6d02f9f01023698"
server_key_version: "encryptionkey@0"
I20260619 18:58:26.963763 3683 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:58:26.968801 3698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:26.971366 3683 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260619 18:58:26.971666 3683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "a65d928d6ec44f7aa74ef53e2904b890"
format_stamp: "Formatted at 2026-06-19 18:58:26 on dist-test-slave-dm3f"
server_key: "a2ec4ae587a580fe22dcd6fed04cff42"
server_key_iv: "ebf045fa0c1ffe49f6d02f9f01023698"
server_key_version: "encryptionkey@0"
I20260619 18:58:26.971942 3683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:26.999483 3683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:27.004732 3683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:27.005119 3683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:27.028201 3683 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35189
I20260619 18:58:27.028317 3749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35189 every 8 connection(s)
I20260619 18:58:27.029852 3683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:27.032317 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 3683
I20260619 18:58:27.032579 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:27.033457 15724 external_mini_cluster.cc:1428] Setting key 88c660cfad8faad408f6fcd4fa66d568
I20260619 18:58:27.035511 3750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:58:27 dist-test-slave-dm3f krb5kdc[3670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:58:27.053254 3750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890: Bootstrap starting.
I20260619 18:58:27.057471 3750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:27.059149 3750 log.cc:824] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:27.062829 3750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890: No bootstrap required, opened a new log
I20260619 18:58:27.065213 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:27.036368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52427 (local address 127.0.0.1:35189)
0619 18:58:27.037463 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:27.037482 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:27.037541 (+ 59us) server_negotiation.cc:407] Connection header received
0619 18:58:27.040187 (+ 2646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:27.040242 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:27.041208 (+ 966us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:27.041913 (+ 705us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:27.043733 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:27.046038 (+ 2305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:27.047717 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:27.048910 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:27.049168 (+ 258us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:27.054913 (+ 5745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:27.054926 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:27.054946 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:27.055009 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:27.060155 (+ 5146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:27.061266 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:27.061274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:27.061282 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:27.061483 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:27.062027 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:27.062031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:27.062033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:27.062638 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:27.063078 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:27.063666 (+ 588us) server_negotiation.cc:299] Negotiation successful
0619 18:58:27.064038 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":434,"threads_started":1}
I20260619 18:58:27.070963 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:27.073201 3750 raft_consensus.cc:348] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d928d6ec44f7aa74ef53e2904b890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } }
I20260619 18:58:27.073712 3750 raft_consensus.cc:374] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:27.073836 3750 raft_consensus.cc:729] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a65d928d6ec44f7aa74ef53e2904b890, State: Initialized, Role: FOLLOWER
I20260619 18:58:27.074635 3750 consensus_queue.cc:260] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d928d6ec44f7aa74ef53e2904b890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } }
I20260619 18:58:27.074987 3750 raft_consensus.cc:388] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:27.075186 3750 raft_consensus.cc:482] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:27.075385 3750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:27.079010 3750 raft_consensus.cc:504] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d928d6ec44f7aa74ef53e2904b890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } }
I20260619 18:58:27.079838 3750 leader_election.cc:302] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a65d928d6ec44f7aa74ef53e2904b890; no voters:
I20260619 18:58:27.081332 3750 leader_election.cc:288] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:27.081674 3760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:27.083117 3760 raft_consensus.cc:686] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [term 1 LEADER]: Becoming Leader. State: Replica: a65d928d6ec44f7aa74ef53e2904b890, State: Running, Role: LEADER
I20260619 18:58:27.083882 3760 consensus_queue.cc:237] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d928d6ec44f7aa74ef53e2904b890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } }
I20260619 18:58:27.084086 3750 sys_catalog.cc:564] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:27.092047 3762 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a65d928d6ec44f7aa74ef53e2904b890. Latest consensus state: current_term: 1 leader_uuid: "a65d928d6ec44f7aa74ef53e2904b890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d928d6ec44f7aa74ef53e2904b890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } } }
I20260619 18:58:27.092542 3762 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:27.093415 3761 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a65d928d6ec44f7aa74ef53e2904b890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d928d6ec44f7aa74ef53e2904b890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } } }
I20260619 18:58:27.093812 3761 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:27.098225 3768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:27.101282 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:27.076673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41553 (local address 127.0.0.1:35189)
0619 18:58:27.076883 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:27.076890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:27.077912 (+ 1022us) server_negotiation.cc:407] Connection header received
0619 18:58:27.078142 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:27.078148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:27.078230 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:27.078361 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:27.080365 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:27.082214 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:27.084376 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:27.085170 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:27.085341 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:27.090405 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:27.090417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:27.090430 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:27.090509 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:27.096666 (+ 6157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:27.098200 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:27.098211 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:27.098217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:27.098519 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:27.099173 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:27.099178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:27.099180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:27.099599 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:27.099999 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:27.100586 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 18:58:27.100807 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:58:27.108162 3768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:27.120568 3768 catalog_manager.cc:1347] Generated new cluster ID: 2b58d28f4d10448fa08ada9474a6c67b
I20260619 18:58:27.120764 3768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:27.136219 3768 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:27.137754 3768 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:27.152056 3768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a65d928d6ec44f7aa74ef53e2904b890: Generated new TSK 0
I20260619 18:58:27.153295 3768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:27.170027 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 3683
I20260619 18:58:27.171603 3745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:27.270954 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 3683
W20260619 18:58:27.289619 3668 connection.cc:535] client connection to 127.0.0.1:35189 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35189): Connection reset by peer (error 104)
2026-06-19T18:58:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2157 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:27 dist-test-slave-dm3f krb5kdc[3782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:27 dist-test-slave-dm3f krb5kdc[3782](info): set up 2 sockets
Jun 19 18:58:27 dist-test-slave-dm3f krb5kdc[3782](info): commencing operation
krb5kdc: starting...
W20260619 18:58:28.391283 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:28 dist-test-slave-dm3f krb5kdc[3782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:28.666113 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:29.033583 3795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:29.033937 3795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:29.034036 3795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:29.041899 3795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:29.042052 3795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:29.042101 3795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:29.042177 3795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:29.042224 3795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:29.052977 3795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:29.054392 3795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:29.055949 3795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:29.069928 3800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:29.069926 3801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:29.069849 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:29.070706 3795 server_base.cc:1034] running on GCE node
I20260619 18:58:29.071774 3795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:29.073619 3795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:29.074894 3795 hybrid_clock.cc:648] HybridClock initialized: now 1781895509074852 us; error 57 us; skew 500 ppm
I20260619 18:58:29.075359 3795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:29 dist-test-slave-dm3f krb5kdc[3782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:29.079221 3795 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:29.081167 3795 webserver.cc:460] Webserver started at http://127.0.0.1:37813/ using document root <none> and password file <none>
I20260619 18:58:29.082018 3795 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:29.082202 3795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:29.082623 3795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:29.086122 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "f4a16b4fea5946a382bc4158c93f3048"
format_stamp: "Formatted at 2026-06-19 18:58:29 on dist-test-slave-dm3f"
server_key: "6af671c5876253001478507add2b81db"
server_key_iv: "b52445bc8162c90c8e0e74355cbf2c8e"
server_key_version: "encryptionkey@0"
I20260619 18:58:29.087227 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "f4a16b4fea5946a382bc4158c93f3048"
format_stamp: "Formatted at 2026-06-19 18:58:29 on dist-test-slave-dm3f"
server_key: "6af671c5876253001478507add2b81db"
server_key_iv: "b52445bc8162c90c8e0e74355cbf2c8e"
server_key_version: "encryptionkey@0"
I20260619 18:58:29.093703 3795 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:58:29.098212 3810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:29.100028 3795 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:29.100347 3795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "f4a16b4fea5946a382bc4158c93f3048"
format_stamp: "Formatted at 2026-06-19 18:58:29 on dist-test-slave-dm3f"
server_key: "6af671c5876253001478507add2b81db"
server_key_iv: "b52445bc8162c90c8e0e74355cbf2c8e"
server_key_version: "encryptionkey@0"
I20260619 18:58:29.100628 3795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:29.142565 3795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:29.147562 3795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:29.147925 3795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:29.169982 3795 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36403
I20260619 18:58:29.170003 3861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36403 every 8 connection(s)
I20260619 18:58:29.171728 3795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:29.174810 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 3795
I20260619 18:58:29.175014 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:29.175485 15724 external_mini_cluster.cc:1428] Setting key 40dc5befad48792a3e527a50f701abf1
I20260619 18:58:29.176681 3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:29.191689 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:29.177659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34053 (local address 127.0.0.1:36403)
0619 18:58:29.178696 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:29.178741 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:29.179111 (+ 370us) server_negotiation.cc:407] Connection header received
0619 18:58:29.181068 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:29.181100 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:29.181664 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:29.182175 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:29.182924 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:29.184450 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:29.185954 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:29.186881 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:29.187114 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:29.188710 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:29.188722 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:29.188745 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:29.189489 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:29.189601 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:29.189635 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:29.189859 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:29.190231 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 18:58:29.190439 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":444,"threads_started":1}
I20260619 18:58:29.191998 3862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048: Bootstrap starting.
I20260619 18:58:29.195833 3862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:29.196447 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:29.197252 3862 log.cc:824] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:29.200742 3862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048: No bootstrap required, opened a new log
I20260619 18:58:29.209892 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:29.201389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35455 (local address 127.0.0.1:36403)
0619 18:58:29.201660 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:29.201669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:29.202118 (+ 449us) server_negotiation.cc:407] Connection header received
0619 18:58:29.202434 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:29.202438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:29.202512 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:29.202610 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:29.203364 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:29.204233 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:29.206522 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:29.207156 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:29.207292 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:29.208844 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:29.208849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:29.208852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:29.208876 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:29.208931 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:29.208949 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:29.209067 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:29.209324 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 18:58:29.209491 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 18:58:29.212393 3862 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a16b4fea5946a382bc4158c93f3048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36403 } }
I20260619 18:58:29.212894 3862 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:29.213027 3862 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a16b4fea5946a382bc4158c93f3048, State: Initialized, Role: FOLLOWER
I20260619 18:58:29.213690 3862 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a16b4fea5946a382bc4158c93f3048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36403 } }
I20260619 18:58:29.213990 3862 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:29.214154 3862 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:29.214365 3862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:29.217186 3862 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a16b4fea5946a382bc4158c93f3048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36403 } }
I20260619 18:58:29.217905 3862 leader_election.cc:302] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4a16b4fea5946a382bc4158c93f3048; no voters:
I20260619 18:58:29.218618 3862 leader_election.cc:288] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:29.218859 3872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:29.219650 3872 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [term 1 LEADER]: Becoming Leader. State: Replica: f4a16b4fea5946a382bc4158c93f3048, State: Running, Role: LEADER
I20260619 18:58:29.220276 3872 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a16b4fea5946a382bc4158c93f3048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36403 } }
I20260619 18:58:29.221237 3862 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:29.224217 3873 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4a16b4fea5946a382bc4158c93f3048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a16b4fea5946a382bc4158c93f3048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36403 } } }
I20260619 18:58:29.224195 3874 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a16b4fea5946a382bc4158c93f3048. Latest consensus state: current_term: 1 leader_uuid: "f4a16b4fea5946a382bc4158c93f3048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a16b4fea5946a382bc4158c93f3048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36403 } } }
I20260619 18:58:29.224753 3873 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:29.224777 3874 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:29.227315 3879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:29.234409 3879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:29.245890 3879 catalog_manager.cc:1347] Generated new cluster ID: b446d6f217c74b3c8496ca3974a12ebd
I20260619 18:58:29.246030 3879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:29.265784 3879 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:29.267997 3879 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:29.278436 3879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4a16b4fea5946a382bc4158c93f3048: Generated new TSK 0
I20260619 18:58:29.279845 3879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:29.287009 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 3795
W20260619 18:58:29.288983 3857 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34053
W20260619 18:58:29.290097 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 3795: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:58:29.290350 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 3795
W20260619 18:58:29.310423 3780 connection.cc:535] client connection to 127.0.0.1:36403 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36403): Connection reset by peer (error 104)
2026-06-19T18:58:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:29 dist-test-slave-dm3f krb5kdc[3893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:29 dist-test-slave-dm3f krb5kdc[3893](info): set up 2 sockets
Jun 19 18:58:29 dist-test-slave-dm3f krb5kdc[3893](info): commencing operation
krb5kdc: starting...
W20260619 18:58:30.419370 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:30 dist-test-slave-dm3f krb5kdc[3893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:30.690371 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:31.060459 3906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:31.060797 3906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:31.060895 3906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:31.068832 3906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:31.068971 3906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:31.069020 3906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:31.069088 3906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:31.069134 3906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:31.080045 3906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:31.081565 3906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:31.083104 3906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:31.096482 3911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:31.096554 3914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:31.096575 3912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:31.097421 3906 server_base.cc:1034] running on GCE node
I20260619 18:58:31.098464 3906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:31.100236 3906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:31.101536 3906 hybrid_clock.cc:648] HybridClock initialized: now 1781895511101489 us; error 64 us; skew 500 ppm
I20260619 18:58:31.101987 3906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:31 dist-test-slave-dm3f krb5kdc[3893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:31.105713 3906 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:31.107725 3906 webserver.cc:460] Webserver started at http://127.0.0.1:40269/ using document root <none> and password file <none>
I20260619 18:58:31.108552 3906 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:31.108737 3906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:31.109089 3906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:31.112663 3906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "87f3523139504b6a9062b2fbfd602284"
format_stamp: "Formatted at 2026-06-19 18:58:31 on dist-test-slave-dm3f"
server_key: "cc8127d6f627bab29f22051ad05605fe"
server_key_iv: "86adefe7cd1da5bb1580ca5033facd20"
server_key_version: "encryptionkey@0"
I20260619 18:58:31.113659 3906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "87f3523139504b6a9062b2fbfd602284"
format_stamp: "Formatted at 2026-06-19 18:58:31 on dist-test-slave-dm3f"
server_key: "cc8127d6f627bab29f22051ad05605fe"
server_key_iv: "86adefe7cd1da5bb1580ca5033facd20"
server_key_version: "encryptionkey@0"
I20260619 18:58:31.120270 3906 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:58:31.124545 3921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:31.126199 3906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:58:31.126469 3906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "87f3523139504b6a9062b2fbfd602284"
format_stamp: "Formatted at 2026-06-19 18:58:31 on dist-test-slave-dm3f"
server_key: "cc8127d6f627bab29f22051ad05605fe"
server_key_iv: "86adefe7cd1da5bb1580ca5033facd20"
server_key_version: "encryptionkey@0"
I20260619 18:58:31.126760 3906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:31.133733 3906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:31.139412 3906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:31.139779 3906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:31.160583 3906 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38465
I20260619 18:58:31.160624 3972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38465 every 8 connection(s)
I20260619 18:58:31.162289 3906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:31.166693 3973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:31.168999 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 3906
I20260619 18:58:31.169219 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:31.169699 15724 external_mini_cluster.cc:1428] Setting key e6ab0dfcdc0d9098b5082f30fa7c2fd4
I20260619 18:58:31.178963 3976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:31.171815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37367 (local address 127.0.0.1:38465)
0619 18:58:31.172762 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:31.172777 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:31.172829 (+ 52us) server_negotiation.cc:407] Connection header received
0619 18:58:31.174641 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:31.174677 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:31.175235 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:31.175714 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:31.176991 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:31.176998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:31.177013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:31.177032 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:31.177117 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:31.177147 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:31.177412 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:31.177580 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 18:58:31.177797 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":422,"threads_started":1}
I20260619 18:58:31.182924 3973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284: Bootstrap starting.
I20260619 18:58:31.183784 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:31.186650 3973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:31.187942 3973 log.cc:824] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:31.191795 3976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:31.188464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52721 (local address 127.0.0.1:38465)
0619 18:58:31.188732 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:31.188743 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:31.189475 (+ 732us) server_negotiation.cc:407] Connection header received
0619 18:58:31.189714 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:31.189720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:31.189782 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:31.189915 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:31.191103 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:31.191109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:31.191113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:31.191145 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:31.191203 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:31.191226 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:31.191370 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:31.191429 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 18:58:31.191496 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:58:31.192095 3973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284: No bootstrap required, opened a new log
I20260619 18:58:31.200736 3973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f3523139504b6a9062b2fbfd602284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } }
I20260619 18:58:31.201190 3973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:31.201330 3973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f3523139504b6a9062b2fbfd602284, State: Initialized, Role: FOLLOWER
I20260619 18:58:31.202150 3973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f3523139504b6a9062b2fbfd602284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } }
I20260619 18:58:31.202497 3973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:31.202705 3973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:31.202967 3973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:31.205436 3973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f3523139504b6a9062b2fbfd602284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } }
I20260619 18:58:31.206118 3973 leader_election.cc:302] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87f3523139504b6a9062b2fbfd602284; no voters:
I20260619 18:58:31.206943 3973 leader_election.cc:288] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:31.207142 3983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:31.207875 3983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [term 1 LEADER]: Becoming Leader. State: Replica: 87f3523139504b6a9062b2fbfd602284, State: Running, Role: LEADER
I20260619 18:58:31.208580 3983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f3523139504b6a9062b2fbfd602284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } }
I20260619 18:58:31.209434 3973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:31.212392 3984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87f3523139504b6a9062b2fbfd602284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f3523139504b6a9062b2fbfd602284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } } }
I20260619 18:58:31.212937 3984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:31.213578 3985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87f3523139504b6a9062b2fbfd602284. Latest consensus state: current_term: 1 leader_uuid: "87f3523139504b6a9062b2fbfd602284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f3523139504b6a9062b2fbfd602284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } } }
I20260619 18:58:31.213917 3985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:31.214174 3992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:31.221089 3992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:31.232462 3992 catalog_manager.cc:1347] Generated new cluster ID: 6ab326c924e34bbb9941f123204a411d
I20260619 18:58:31.232618 3992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:31.250135 3992 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:31.250701 3992 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:31.258273 3992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87f3523139504b6a9062b2fbfd602284: Generated new TSK 0
I20260619 18:58:31.259388 3992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:31.275027 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 3906
W20260619 18:58:31.276733 3968 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37367
W20260619 18:58:31.277714 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 3906: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:58:31.277902 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 3906
2026-06-19T18:58:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:31 dist-test-slave-dm3f krb5kdc[4004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:31 dist-test-slave-dm3f krb5kdc[4004](info): set up 2 sockets
Jun 19 18:58:31 dist-test-slave-dm3f krb5kdc[4004](info): commencing operation
krb5kdc: starting...
W20260619 18:58:32.407333 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.005s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:32 dist-test-slave-dm3f krb5kdc[4004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:32.686514 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:33.059656 4017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:33.060070 4017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:33.060246 4017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:33.068406 4017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:33.068607 4017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:33.068742 4017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:33.068857 4017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:33.068940 4017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:33.080165 4017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:33.081713 4017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:33.083348 4017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:33.097391 4022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:33.097482 4023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:33.097433 4025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:33.098587 4017 server_base.cc:1034] running on GCE node
I20260619 18:58:33.099594 4017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:33.101347 4017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:33.102689 4017 hybrid_clock.cc:648] HybridClock initialized: now 1781895513102631 us; error 70 us; skew 500 ppm
I20260619 18:58:33.103266 4017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:33 dist-test-slave-dm3f krb5kdc[4004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:33.110563 4017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:33.112582 4017 webserver.cc:460] Webserver started at http://127.0.0.1:43295/ using document root <none> and password file <none>
I20260619 18:58:33.113411 4017 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:33.113551 4017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:33.113839 4017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:33.117509 4017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "570c55070a6a452788eb8930d592757f"
format_stamp: "Formatted at 2026-06-19 18:58:33 on dist-test-slave-dm3f"
server_key: "79ef87f3a8e7bb946a6fc3a721ebc618"
server_key_iv: "d45ed03d9da79531c117c9e72b5c37fc"
server_key_version: "encryptionkey@0"
I20260619 18:58:33.118441 4017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "570c55070a6a452788eb8930d592757f"
format_stamp: "Formatted at 2026-06-19 18:58:33 on dist-test-slave-dm3f"
server_key: "79ef87f3a8e7bb946a6fc3a721ebc618"
server_key_iv: "d45ed03d9da79531c117c9e72b5c37fc"
server_key_version: "encryptionkey@0"
I20260619 18:58:33.124923 4017 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:58:33.129638 4032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:33.131608 4017 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:33.131834 4017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "570c55070a6a452788eb8930d592757f"
format_stamp: "Formatted at 2026-06-19 18:58:33 on dist-test-slave-dm3f"
server_key: "79ef87f3a8e7bb946a6fc3a721ebc618"
server_key_iv: "d45ed03d9da79531c117c9e72b5c37fc"
server_key_version: "encryptionkey@0"
I20260619 18:58:33.132056 4017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:33.154937 4017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:33.160640 4017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:33.160965 4017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:33.183230 4017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39907
I20260619 18:58:33.183369 4083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39907 every 8 connection(s)
I20260619 18:58:33.184978 4017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:33.185307 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 4017
I20260619 18:58:33.185663 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:33.186539 15724 external_mini_cluster.cc:1428] Setting key 53c5add982cd91be4045e98d0bc1ec32
I20260619 18:58:33.192193 4084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:58:33 dist-test-slave-dm3f krb5kdc[4004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:58:33.204977 4084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f: Bootstrap starting.
I20260619 18:58:33.209764 4084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:33.211252 4084 log.cc:824] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:33.215360 4084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f: No bootstrap required, opened a new log
I20260619 18:58:33.215667 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:33.189639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44043 (local address 127.0.0.1:39907)
0619 18:58:33.190862 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:33.190882 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:33.190950 (+ 68us) server_negotiation.cc:407] Connection header received
0619 18:58:33.193114 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:33.193145 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:33.193658 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:33.194150 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:33.195925 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:33.197506 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:33.199051 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:33.199880 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:33.200135 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:33.205136 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:33.205149 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:33.205174 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:33.205239 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:33.209689 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:33.210787 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:33.210798 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:33.210808 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:33.211101 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:33.211888 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:33.211900 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:33.211906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:33.212693 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:33.213094 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:33.213583 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 18:58:33.214116 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":491,"threads_started":1}
I20260619 18:58:33.221485 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:33.225076 4084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c55070a6a452788eb8930d592757f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260619 18:58:33.225584 4084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:33.225726 4084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 570c55070a6a452788eb8930d592757f, State: Initialized, Role: FOLLOWER
I20260619 18:58:33.226403 4084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c55070a6a452788eb8930d592757f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260619 18:58:33.226867 4084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:33.227059 4084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:33.227325 4084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:33.230249 4084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c55070a6a452788eb8930d592757f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260619 18:58:33.231038 4084 leader_election.cc:302] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 570c55070a6a452788eb8930d592757f; no voters:
I20260619 18:58:33.231824 4084 leader_election.cc:288] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:33.231937 4094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:33.233433 4094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [term 1 LEADER]: Becoming Leader. State: Replica: 570c55070a6a452788eb8930d592757f, State: Running, Role: LEADER
I20260619 18:58:33.234256 4094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c55070a6a452788eb8930d592757f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } }
I20260619 18:58:33.234771 4084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:33.238195 4095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "570c55070a6a452788eb8930d592757f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c55070a6a452788eb8930d592757f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } } }
I20260619 18:58:33.238322 4096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 570c55070a6a452788eb8930d592757f. Latest consensus state: current_term: 1 leader_uuid: "570c55070a6a452788eb8930d592757f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570c55070a6a452788eb8930d592757f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39907 } } }
I20260619 18:58:33.238808 4096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:33.238816 4095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:33.244762 4102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:33.253692 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:33.226947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45719 (local address 127.0.0.1:39907)
0619 18:58:33.227212 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:33.227220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:33.227587 (+ 367us) server_negotiation.cc:407] Connection header received
0619 18:58:33.227750 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:33.227759 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:33.227855 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:33.227984 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:33.229601 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:33.230774 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:33.232418 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:33.233164 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:33.233304 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:33.236892 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:33.236900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:33.236905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:33.236965 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:33.246930 (+ 9965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:33.248517 (+ 1587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:33.248527 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:33.248534 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:33.248867 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:33.249539 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:33.249547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:33.249553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:33.250148 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:33.252373 (+ 2225us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:33.252627 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 18:58:33.253036 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:58:33.257479 4102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:33.268893 4102 catalog_manager.cc:1347] Generated new cluster ID: c67dc87be17645119682f6f2f9b7cb0a
I20260619 18:58:33.269037 4102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:33.291074 4102 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:33.293782 4102 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:33.308037 4102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 570c55070a6a452788eb8930d592757f: Generated new TSK 0
I20260619 18:58:33.309067 4102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:33.316978 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4017
I20260619 18:58:33.318444 4079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:33.416581 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 4017
2026-06-19T18:58:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2139 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:33 dist-test-slave-dm3f krb5kdc[4116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:33 dist-test-slave-dm3f krb5kdc[4116](info): set up 2 sockets
Jun 19 18:58:33 dist-test-slave-dm3f krb5kdc[4116](info): commencing operation
krb5kdc: starting...
W20260619 18:58:34.547292 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:34 dist-test-slave-dm3f krb5kdc[4116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:34.826865 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:35.197507 4129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:35.197907 4129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:35.198067 4129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:35.205964 4129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:35.206158 4129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:35.206287 4129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:35.206396 4129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:35.206475 4129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:35.217036 4129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:35.218622 4129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:35.220177 4129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:35.234252 4134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:35.234177 4137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:35.234220 4135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:35.235641 4129 server_base.cc:1034] running on GCE node
I20260619 18:58:35.236799 4129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:35.238521 4129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:35.239828 4129 hybrid_clock.cc:648] HybridClock initialized: now 1781895515239804 us; error 92 us; skew 500 ppm
I20260619 18:58:35.240281 4129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:35 dist-test-slave-dm3f krb5kdc[4116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:35.244256 4129 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:35.246053 4129 webserver.cc:460] Webserver started at http://127.0.0.1:39767/ using document root <none> and password file <none>
I20260619 18:58:35.246969 4129 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:35.247126 4129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:35.247486 4129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:35.251044 4129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "37b83150a646482595e2051c4a5662ea"
format_stamp: "Formatted at 2026-06-19 18:58:35 on dist-test-slave-dm3f"
server_key: "9847f1bc737716faa4f78a3aca50c7f9"
server_key_iv: "77a4e31278916188c6f3fda2e07ed0f8"
server_key_version: "encryptionkey@0"
I20260619 18:58:35.251995 4129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "37b83150a646482595e2051c4a5662ea"
format_stamp: "Formatted at 2026-06-19 18:58:35 on dist-test-slave-dm3f"
server_key: "9847f1bc737716faa4f78a3aca50c7f9"
server_key_iv: "77a4e31278916188c6f3fda2e07ed0f8"
server_key_version: "encryptionkey@0"
I20260619 18:58:35.259392 4129 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260619 18:58:35.263876 4144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:35.265579 4129 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:58:35.265849 4129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "37b83150a646482595e2051c4a5662ea"
format_stamp: "Formatted at 2026-06-19 18:58:35 on dist-test-slave-dm3f"
server_key: "9847f1bc737716faa4f78a3aca50c7f9"
server_key_iv: "77a4e31278916188c6f3fda2e07ed0f8"
server_key_version: "encryptionkey@0"
I20260619 18:58:35.266076 4129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:35.319316 4129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:35.324640 4129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:35.325016 4129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:35.347468 4129 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44693
I20260619 18:58:35.347499 4195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44693 every 8 connection(s)
I20260619 18:58:35.349143 4129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:35.353706 4196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:35.356961 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 4129
I20260619 18:58:35.357146 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:35.357643 15724 external_mini_cluster.cc:1428] Setting key b26ddb96595d3cd08edda010e07aedd3
I20260619 18:58:35.367046 4196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea: Bootstrap starting.
I20260619 18:58:35.371081 4196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:35.372488 4196 log.cc:824] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:35.373706 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:35.359833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46239 (local address 127.0.0.1:44693)
0619 18:58:35.360784 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:35.360798 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:35.360849 (+ 51us) server_negotiation.cc:407] Connection header received
0619 18:58:35.362892 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:35.362923 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:35.363459 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:35.363983 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:35.364688 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:35.366108 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:35.367724 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:35.368531 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:35.368829 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:35.370233 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:35.370244 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:35.370263 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:35.370988 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:35.371098 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:35.371131 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:35.371296 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:35.371610 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 18:58:35.372115 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":442,"threads_started":1}
I20260619 18:58:35.376511 4196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea: No bootstrap required, opened a new log
I20260619 18:58:35.378813 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:35.388212 4196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b83150a646482595e2051c4a5662ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } }
I20260619 18:58:35.388820 4196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:35.388970 4196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b83150a646482595e2051c4a5662ea, State: Initialized, Role: FOLLOWER
I20260619 18:58:35.389835 4196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b83150a646482595e2051c4a5662ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } }
I20260619 18:58:35.390210 4196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:35.390368 4196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:35.390553 4196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:35.394088 4196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b83150a646482595e2051c4a5662ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } }
I20260619 18:58:35.394666 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:35.386163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52779 (local address 127.0.0.1:44693)
0619 18:58:35.386410 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:35.386417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:35.387098 (+ 681us) server_negotiation.cc:407] Connection header received
0619 18:58:35.387297 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:35.387303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:35.387401 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:35.387525 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:35.388337 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:35.389498 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:35.391644 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:35.392347 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:35.392484 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:35.393826 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:35.393832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:35.393838 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:35.393862 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:35.393911 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:35.393929 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:35.394013 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:35.394215 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 18:58:35.394308 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 18:58:35.394963 4196 leader_election.cc:302] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37b83150a646482595e2051c4a5662ea; no voters:
I20260619 18:58:35.395722 4196 leader_election.cc:288] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:35.395843 4206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:35.397898 4196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:35.399103 4206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [term 1 LEADER]: Becoming Leader. State: Replica: 37b83150a646482595e2051c4a5662ea, State: Running, Role: LEADER
I20260619 18:58:35.399912 4206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b83150a646482595e2051c4a5662ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } }
I20260619 18:58:35.403590 4207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37b83150a646482595e2051c4a5662ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b83150a646482595e2051c4a5662ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } } }
I20260619 18:58:35.403492 4208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37b83150a646482595e2051c4a5662ea. Latest consensus state: current_term: 1 leader_uuid: "37b83150a646482595e2051c4a5662ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b83150a646482595e2051c4a5662ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } } }
I20260619 18:58:35.404098 4207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:35.404106 4208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:35.405272 4220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:35.411415 4220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:35.423179 4220 catalog_manager.cc:1347] Generated new cluster ID: 99130cf930b04bb98843094856f88afa
I20260619 18:58:35.423323 4220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:35.434948 4220 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:35.436417 4220 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:35.449747 4220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37b83150a646482595e2051c4a5662ea: Generated new TSK 0
I20260619 18:58:35.451013 4220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:35.466046 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4129
W20260619 18:58:35.467494 4191 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46239
W20260619 18:58:35.468551 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4129: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:58:35.468789 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 4129
2026-06-19T18:58:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2052 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:35 dist-test-slave-dm3f krb5kdc[4227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:35 dist-test-slave-dm3f krb5kdc[4227](info): set up 2 sockets
Jun 19 18:58:35 dist-test-slave-dm3f krb5kdc[4227](info): commencing operation
krb5kdc: starting...
W20260619 18:58:36.583454 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:36 dist-test-slave-dm3f krb5kdc[4227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:36.861886 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:37.233743 4240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:37.234174 4240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:37.234360 4240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:37.242307 4240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:37.242522 4240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:37.242655 4240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:37.242801 4240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:37.242900 4240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:37.253818 4240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:37.255394 4240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:37.256991 4240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:37.271219 4246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:37.271908 4248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:37.272104 4245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:37.272104 4240 server_base.cc:1034] running on GCE node
I20260619 18:58:37.273159 4240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:37.275000 4240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:37.276332 4240 hybrid_clock.cc:648] HybridClock initialized: now 1781895517276277 us; error 46 us; skew 500 ppm
I20260619 18:58:37.276823 4240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:37 dist-test-slave-dm3f krb5kdc[4227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:37.280531 4240 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:37.282524 4240 webserver.cc:460] Webserver started at http://127.0.0.1:34417/ using document root <none> and password file <none>
I20260619 18:58:37.283401 4240 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:37.283533 4240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:37.283830 4240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:37.287348 4240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "3ea24cb8126a480699e836b64b961d50"
format_stamp: "Formatted at 2026-06-19 18:58:37 on dist-test-slave-dm3f"
server_key: "259034edbf2c0243411d64e1a8f7dcb5"
server_key_iv: "f661b438bd94561b9a1738857464789f"
server_key_version: "encryptionkey@0"
I20260619 18:58:37.288296 4240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "3ea24cb8126a480699e836b64b961d50"
format_stamp: "Formatted at 2026-06-19 18:58:37 on dist-test-slave-dm3f"
server_key: "259034edbf2c0243411d64e1a8f7dcb5"
server_key_iv: "f661b438bd94561b9a1738857464789f"
server_key_version: "encryptionkey@0"
I20260619 18:58:37.294615 4240 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 18:58:37.299357 4255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:37.301705 4240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:58:37.302157 4240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "3ea24cb8126a480699e836b64b961d50"
format_stamp: "Formatted at 2026-06-19 18:58:37 on dist-test-slave-dm3f"
server_key: "259034edbf2c0243411d64e1a8f7dcb5"
server_key_iv: "f661b438bd94561b9a1738857464789f"
server_key_version: "encryptionkey@0"
I20260619 18:58:37.302505 4240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:37.310660 4240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:37.317585 4240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:37.318041 4240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:37.343398 4240 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42931
I20260619 18:58:37.343408 4306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42931 every 8 connection(s)
I20260619 18:58:37.345232 4240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:37.349852 4307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:37.351305 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 4240
I20260619 18:58:37.351576 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:37.352145 15724 external_mini_cluster.cc:1428] Setting key 0fba1ec7950628696b374ecb82ddf69f
I20260619 18:58:37.362581 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:37.354809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49467 (local address 127.0.0.1:42931)
0619 18:58:37.355899 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:37.355914 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:37.355974 (+ 60us) server_negotiation.cc:407] Connection header received
0619 18:58:37.357935 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:37.357972 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:37.358505 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:37.359143 (+ 638us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:37.360424 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:37.360432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:37.360450 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:37.360469 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:37.360540 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:37.360565 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:37.360854 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:37.361023 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 18:58:37.361246 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":511,"threads_started":1}
I20260619 18:58:37.367591 4307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50: Bootstrap starting.
I20260619 18:58:37.368552 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:37.371646 4307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:37.374212 4307 log.cc:824] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:37.376670 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:37.373861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48639 (local address 127.0.0.1:42931)
0619 18:58:37.374095 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:37.374116 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:37.374345 (+ 229us) server_negotiation.cc:407] Connection header received
0619 18:58:37.374620 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:37.374625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:37.374681 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:37.374853 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:37.375966 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:37.375970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:37.375973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:37.375993 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:37.376035 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:37.376052 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:37.376203 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:37.376292 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 18:58:37.376367 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:58:37.378175 4307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50: No bootstrap required, opened a new log
I20260619 18:58:37.387548 4307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea24cb8126a480699e836b64b961d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } }
I20260619 18:58:37.388234 4307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:37.388388 4307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea24cb8126a480699e836b64b961d50, State: Initialized, Role: FOLLOWER
I20260619 18:58:37.389281 4307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea24cb8126a480699e836b64b961d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } }
I20260619 18:58:37.389621 4307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:37.389798 4307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:37.389997 4307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:37.393707 4307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea24cb8126a480699e836b64b961d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } }
I20260619 18:58:37.394392 4307 leader_election.cc:302] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ea24cb8126a480699e836b64b961d50; no voters:
I20260619 18:58:37.395205 4307 leader_election.cc:288] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:37.395402 4317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:37.396821 4317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [term 1 LEADER]: Becoming Leader. State: Replica: 3ea24cb8126a480699e836b64b961d50, State: Running, Role: LEADER
I20260619 18:58:37.397571 4317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea24cb8126a480699e836b64b961d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } }
I20260619 18:58:37.397881 4307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:37.401403 4318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ea24cb8126a480699e836b64b961d50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea24cb8126a480699e836b64b961d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } } }
I20260619 18:58:37.402069 4318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:37.402977 4319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ea24cb8126a480699e836b64b961d50. Latest consensus state: current_term: 1 leader_uuid: "3ea24cb8126a480699e836b64b961d50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea24cb8126a480699e836b64b961d50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } } }
I20260619 18:58:37.403457 4319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:37.410399 4326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:37.417326 4326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:37.428582 4326 catalog_manager.cc:1347] Generated new cluster ID: 40ff4bfcb47e43dabde6dcc798d63e47
I20260619 18:58:37.428766 4326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:37.471696 4326 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:37.472355 4326 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:37.484406 4326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ea24cb8126a480699e836b64b961d50: Generated new TSK 0
I20260619 18:58:37.485668 4326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:37.502182 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4240
W20260619 18:58:37.503716 4302 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49467
W20260619 18:58:37.504680 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4240: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:58:37.504873 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 4240
2026-06-19T18:58:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2035 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:37 dist-test-slave-dm3f krb5kdc[4338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:37 dist-test-slave-dm3f krb5kdc[4338](info): set up 2 sockets
Jun 19 18:58:37 dist-test-slave-dm3f krb5kdc[4338](info): commencing operation
krb5kdc: starting...
W20260619 18:58:38.647370 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:38 dist-test-slave-dm3f krb5kdc[4338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:38.926357 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:39.294166 4351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:39.294507 4351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:39.294607 4351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:39.302556 4351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:39.302695 4351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:39.302799 4351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:39.302887 4351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:39.302935 4351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:39.313544 4351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:39.315004 4351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:39.316424 4351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:39.330101 4357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:39.330163 4359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:39.330194 4356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:39.331147 4351 server_base.cc:1034] running on GCE node
I20260619 18:58:39.332078 4351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:39.333971 4351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:39.335273 4351 hybrid_clock.cc:648] HybridClock initialized: now 1781895519335218 us; error 56 us; skew 500 ppm
I20260619 18:58:39.335716 4351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:39 dist-test-slave-dm3f krb5kdc[4338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:39.343004 4351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:39.344954 4351 webserver.cc:460] Webserver started at http://127.0.0.1:43419/ using document root <none> and password file <none>
I20260619 18:58:39.345759 4351 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:39.345901 4351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:39.346186 4351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:39.349691 4351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "d9ebdecd6fba4a69a4af734446756cfa"
format_stamp: "Formatted at 2026-06-19 18:58:39 on dist-test-slave-dm3f"
server_key: "6ae943aab5763be0b3073ff5d2c8d432"
server_key_iv: "f845b1bda0e5f11f8ffaed8c53bf3818"
server_key_version: "encryptionkey@0"
I20260619 18:58:39.350589 4351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "d9ebdecd6fba4a69a4af734446756cfa"
format_stamp: "Formatted at 2026-06-19 18:58:39 on dist-test-slave-dm3f"
server_key: "6ae943aab5763be0b3073ff5d2c8d432"
server_key_iv: "f845b1bda0e5f11f8ffaed8c53bf3818"
server_key_version: "encryptionkey@0"
I20260619 18:58:39.357133 4351 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:58:39.361591 4366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:39.363642 4351 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:39.363881 4351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "d9ebdecd6fba4a69a4af734446756cfa"
format_stamp: "Formatted at 2026-06-19 18:58:39 on dist-test-slave-dm3f"
server_key: "6ae943aab5763be0b3073ff5d2c8d432"
server_key_iv: "f845b1bda0e5f11f8ffaed8c53bf3818"
server_key_version: "encryptionkey@0"
I20260619 18:58:39.364104 4351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:39.383579 4351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:39.390372 4351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:39.390666 4351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:39.412205 4351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37557
I20260619 18:58:39.412241 4417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37557 every 8 connection(s)
I20260619 18:58:39.413841 4351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:39.415119 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 4351
I20260619 18:58:39.415345 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:39.415995 15724 external_mini_cluster.cc:1428] Setting key 40c369809f5c11ca992d15dff8e2fe18
I20260619 18:58:39.419757 4418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:58:39 dist-test-slave-dm3f krb5kdc[4338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:58:39.432737 4418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa: Bootstrap starting.
I20260619 18:58:39.437011 4418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:39.438326 4418 log.cc:824] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:39.442260 4418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa: No bootstrap required, opened a new log
I20260619 18:58:39.442530 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:39.418182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:57105 (local address 127.0.0.1:37557)
0619 18:58:39.419196 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:39.419210 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:39.419257 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:58:39.421120 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:39.421148 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:39.421631 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:39.422114 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:39.423830 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:39.425377 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:39.426979 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:39.427812 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:39.428066 (+ 254us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:39.433072 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:39.433083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:39.433113 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:39.433165 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:39.436760 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:39.437957 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:39.437964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:39.437973 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:39.438193 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:39.438829 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:39.438833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:39.438835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:39.439793 (+ 958us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:39.440176 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:39.440824 (+ 648us) server_negotiation.cc:299] Negotiation successful
0619 18:58:39.441293 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":454,"threads_started":1}
I20260619 18:58:39.447927 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:39.452148 4418 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ebdecd6fba4a69a4af734446756cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260619 18:58:39.452805 4418 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:39.452966 4418 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ebdecd6fba4a69a4af734446756cfa, State: Initialized, Role: FOLLOWER
I20260619 18:58:39.453774 4418 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ebdecd6fba4a69a4af734446756cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260619 18:58:39.454077 4418 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:39.454289 4418 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:39.454497 4418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:39.457165 4418 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ebdecd6fba4a69a4af734446756cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260619 18:58:39.457854 4418 leader_election.cc:302] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9ebdecd6fba4a69a4af734446756cfa; no voters:
I20260619 18:58:39.458683 4418 leader_election.cc:288] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:39.458877 4428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:39.460281 4428 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [term 1 LEADER]: Becoming Leader. State: Replica: d9ebdecd6fba4a69a4af734446756cfa, State: Running, Role: LEADER
I20260619 18:58:39.461390 4428 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ebdecd6fba4a69a4af734446756cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260619 18:58:39.461759 4418 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:39.465375 4429 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9ebdecd6fba4a69a4af734446756cfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ebdecd6fba4a69a4af734446756cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } } }
I20260619 18:58:39.465869 4429 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:39.467293 4430 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9ebdecd6fba4a69a4af734446756cfa. Latest consensus state: current_term: 1 leader_uuid: "d9ebdecd6fba4a69a4af734446756cfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ebdecd6fba4a69a4af734446756cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } } }
I20260619 18:58:39.467814 4430 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:39.476811 4435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:39.478427 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:39.452872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:41561 (local address 127.0.0.1:37557)
0619 18:58:39.453202 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:39.453210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:39.453659 (+ 449us) server_negotiation.cc:407] Connection header received
0619 18:58:39.453904 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:39.453908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:39.454013 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:39.454151 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:39.455963 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:39.456809 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:39.458444 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:39.459196 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:39.459404 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:39.463058 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:39.463065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:39.463068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:39.463110 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:39.471233 (+ 8123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:39.473048 (+ 1815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:39.473058 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:39.473064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:39.473343 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:39.474226 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:39.474234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:39.474241 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:39.474875 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:39.477139 (+ 2264us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:39.477432 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 18:58:39.477755 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 18:58:39.484440 4435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:39.495752 4435 catalog_manager.cc:1347] Generated new cluster ID: 23e077529d0c4be3a62baef27e579858
I20260619 18:58:39.495898 4435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:39.537761 4435 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:39.539326 4435 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:39.546783 4435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9ebdecd6fba4a69a4af734446756cfa: Generated new TSK 0
I20260619 18:58:39.547981 4435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:39.566851 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4351
I20260619 18:58:39.568267 4413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:39.665812 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 4351
W20260619 18:58:39.684527 4336 connection.cc:535] client connection to 127.0.0.1:37557 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37557): Connection reset by peer (error 104)
2026-06-19T18:58:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2161 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:39 dist-test-slave-dm3f krb5kdc[4450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:39 dist-test-slave-dm3f krb5kdc[4450](info): set up 2 sockets
Jun 19 18:58:39 dist-test-slave-dm3f krb5kdc[4450](info): commencing operation
krb5kdc: starting...
W20260619 18:58:40.791420 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:40 dist-test-slave-dm3f krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:41.079057 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:41.453365 4463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:41.453742 4463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:41.453848 4463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:41.462218 4463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:41.462388 4463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:41.462438 4463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:41.462493 4463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:41.462543 4463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:41.473937 4463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:41.475446 4463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:41.477077 4463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:41.491256 4471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:41.491343 4468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:41.491299 4469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:41.492810 4463 server_base.cc:1034] running on GCE node
I20260619 18:58:41.493798 4463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:41.495674 4463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:41.496980 4463 hybrid_clock.cc:648] HybridClock initialized: now 1781895521496931 us; error 37 us; skew 500 ppm
I20260619 18:58:41.497498 4463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:41 dist-test-slave-dm3f krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:41.502008 4463 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:41.507239 4463 webserver.cc:460] Webserver started at http://127.0.0.1:40397/ using document root <none> and password file <none>
I20260619 18:58:41.508450 4463 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:41.508715 4463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:41.509141 4463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:41.514089 4463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "6f605ab727564e86bc2a9728ea0ac4b9"
format_stamp: "Formatted at 2026-06-19 18:58:41 on dist-test-slave-dm3f"
server_key: "a35d2fec6cce489912eb85b3d5361ad6"
server_key_iv: "fc16e74f37a700da1c03324a760e6f7a"
server_key_version: "encryptionkey@0"
I20260619 18:58:41.515524 4463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "6f605ab727564e86bc2a9728ea0ac4b9"
format_stamp: "Formatted at 2026-06-19 18:58:41 on dist-test-slave-dm3f"
server_key: "a35d2fec6cce489912eb85b3d5361ad6"
server_key_iv: "fc16e74f37a700da1c03324a760e6f7a"
server_key_version: "encryptionkey@0"
I20260619 18:58:41.523380 4463 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 18:58:41.528134 4478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:41.529985 4463 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260619 18:58:41.530275 4463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "6f605ab727564e86bc2a9728ea0ac4b9"
format_stamp: "Formatted at 2026-06-19 18:58:41 on dist-test-slave-dm3f"
server_key: "a35d2fec6cce489912eb85b3d5361ad6"
server_key_iv: "fc16e74f37a700da1c03324a760e6f7a"
server_key_version: "encryptionkey@0"
I20260619 18:58:41.530541 4463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:41.556951 4463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:41.561951 4463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:41.562323 4463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:41.584020 4463 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43167
I20260619 18:58:41.584041 4529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43167 every 8 connection(s)
I20260619 18:58:41.585700 4463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:41.588583 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 4463
I20260619 18:58:41.588804 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:41.589370 15724 external_mini_cluster.cc:1428] Setting key 897705c646e462b338c1af99ff1c30fc
I20260619 18:58:41.592476 4530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:41.604650 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:41.591326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:54093 (local address 127.0.0.1:43167)
0619 18:58:41.592150 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:41.592166 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:41.592216 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:58:41.594121 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:41.594151 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:41.594655 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:41.595189 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:41.595871 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:41.597351 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:41.598887 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:41.599750 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:41.600014 (+ 264us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:41.601573 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:41.601582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:41.601602 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:41.602313 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:41.602417 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:41.602454 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:41.602715 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:41.603195 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 18:58:41.603437 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":345,"threads_started":1}
I20260619 18:58:41.606077 4530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9: Bootstrap starting.
I20260619 18:58:41.609970 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:41.610592 4530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:41.612063 4530 log.cc:824] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:41.617787 4530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9: No bootstrap required, opened a new log
I20260619 18:58:41.631155 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:41.620988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:58739 (local address 127.0.0.1:43167)
0619 18:58:41.621226 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:41.621236 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:41.621925 (+ 689us) server_negotiation.cc:407] Connection header received
0619 18:58:41.622259 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:41.622270 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:41.622399 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:41.622520 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:41.623371 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:41.624928 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:41.627269 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:41.628305 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:41.628480 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:41.630031 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:41.630063 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:41.630070 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:41.630099 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:41.630163 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:41.630201 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:41.630402 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:41.630672 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 18:58:41.630786 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:58:41.633075 4530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f605ab727564e86bc2a9728ea0ac4b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260619 18:58:41.633705 4530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:41.633857 4530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f605ab727564e86bc2a9728ea0ac4b9, State: Initialized, Role: FOLLOWER
I20260619 18:58:41.634652 4530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f605ab727564e86bc2a9728ea0ac4b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260619 18:58:41.635013 4530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:41.635208 4530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:41.635409 4530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:41.638449 4530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f605ab727564e86bc2a9728ea0ac4b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260619 18:58:41.639189 4530 leader_election.cc:302] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f605ab727564e86bc2a9728ea0ac4b9; no voters:
I20260619 18:58:41.639992 4530 leader_election.cc:288] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:41.640172 4540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:41.641721 4540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [term 1 LEADER]: Becoming Leader. State: Replica: 6f605ab727564e86bc2a9728ea0ac4b9, State: Running, Role: LEADER
I20260619 18:58:41.642295 4540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f605ab727564e86bc2a9728ea0ac4b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260619 18:58:41.642782 4530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:41.646503 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f605ab727564e86bc2a9728ea0ac4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f605ab727564e86bc2a9728ea0ac4b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } } }
I20260619 18:58:41.646552 4542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f605ab727564e86bc2a9728ea0ac4b9. Latest consensus state: current_term: 1 leader_uuid: "6f605ab727564e86bc2a9728ea0ac4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f605ab727564e86bc2a9728ea0ac4b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } } }
I20260619 18:58:41.647071 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:41.647101 4542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:41.649801 4546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:41.657068 4546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:41.670310 4546 catalog_manager.cc:1347] Generated new cluster ID: fb5d343d0e7f4bfc8a22c3390fa2f602
I20260619 18:58:41.670464 4546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:41.706270 4546 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:41.708473 4546 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:41.717461 4546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f605ab727564e86bc2a9728ea0ac4b9: Generated new TSK 0
I20260619 18:58:41.719194 4546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:41.727941 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4463
W20260619 18:58:41.729321 4525 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:54093
W20260619 18:58:41.730441 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4463: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:58:41.730647 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 4463
W20260619 18:58:41.749617 4448 connection.cc:535] client connection to 127.0.0.1:43167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43167): Connection reset by peer (error 104)
2026-06-19T18:58:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2064 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:41 dist-test-slave-dm3f krb5kdc[4561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:41 dist-test-slave-dm3f krb5kdc[4561](info): set up 2 sockets
Jun 19 18:58:41 dist-test-slave-dm3f krb5kdc[4561](info): commencing operation
krb5kdc: starting...
W20260619 18:58:42.859437 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:43 dist-test-slave-dm3f krb5kdc[4561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:43.141624 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:43.508905 4574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:43.509227 4574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:43.509338 4574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:43.520244 4574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:43.520404 4574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:43.520454 4574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:43.520512 4574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:43.520560 4574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:43.531656 4574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:43.533059 4574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:43.534543 4574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:43.548231 4580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:43.548231 4582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:43.548883 4579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:43.549568 4574 server_base.cc:1034] running on GCE node
I20260619 18:58:43.550452 4574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:43.552194 4574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:43.553449 4574 hybrid_clock.cc:648] HybridClock initialized: now 1781895523553385 us; error 39 us; skew 500 ppm
I20260619 18:58:43.554008 4574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:43 dist-test-slave-dm3f krb5kdc[4561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:43.557639 4574 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:43.559691 4574 webserver.cc:460] Webserver started at http://127.0.0.1:45481/ using document root <none> and password file <none>
I20260619 18:58:43.560741 4574 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:43.560976 4574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:43.561408 4574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:43.566820 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "e813a8b64ea04f57965852fecbf55732"
format_stamp: "Formatted at 2026-06-19 18:58:43 on dist-test-slave-dm3f"
server_key: "19a57343eac1dabe9cf7c716aa2c7de9"
server_key_iv: "cf489811444ccf3822b06d38bab0f4b2"
server_key_version: "encryptionkey@0"
I20260619 18:58:43.568219 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "e813a8b64ea04f57965852fecbf55732"
format_stamp: "Formatted at 2026-06-19 18:58:43 on dist-test-slave-dm3f"
server_key: "19a57343eac1dabe9cf7c716aa2c7de9"
server_key_iv: "cf489811444ccf3822b06d38bab0f4b2"
server_key_version: "encryptionkey@0"
I20260619 18:58:43.576911 4574 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260619 18:58:43.581278 4589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:43.583063 4574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:43.583315 4574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "e813a8b64ea04f57965852fecbf55732"
format_stamp: "Formatted at 2026-06-19 18:58:43 on dist-test-slave-dm3f"
server_key: "19a57343eac1dabe9cf7c716aa2c7de9"
server_key_iv: "cf489811444ccf3822b06d38bab0f4b2"
server_key_version: "encryptionkey@0"
I20260619 18:58:43.583556 4574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:43.590130 4574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:43.595124 4574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:43.595477 4574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:43.616384 4574 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36101
I20260619 18:58:43.616454 4640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36101 every 8 connection(s)
I20260619 18:58:43.618201 4574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:43.619812 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 4574
I20260619 18:58:43.620023 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:43.620476 15724 external_mini_cluster.cc:1428] Setting key 338f5969c0ebf094b6dded3c800657c3
I20260619 18:58:43.623534 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:43.630760 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:43.622675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:36037 (local address 127.0.0.1:36101)
0619 18:58:43.623676 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:43.623694 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:43.623752 (+ 58us) server_negotiation.cc:407] Connection header received
0619 18:58:43.626183 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:43.626215 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:43.626744 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:43.627294 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:43.628422 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:43.628431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:43.628452 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:43.628480 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:43.628591 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:43.628622 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:43.628873 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:43.629031 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 18:58:43.629301 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":370,"threads_started":1}
I20260619 18:58:43.635792 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:43.639814 4641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732: Bootstrap starting.
I20260619 18:58:43.644475 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:43.641583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:39783 (local address 127.0.0.1:36101)
0619 18:58:43.641821 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:43.641828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:43.642185 (+ 357us) server_negotiation.cc:407] Connection header received
0619 18:58:43.642361 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:43.642366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:43.642446 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:43.642569 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:43.643671 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:43.643677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:43.643682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:43.643703 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:43.643758 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:43.643779 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:43.643913 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:43.644020 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 18:58:43.644090 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:58:43.644479 4641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:43.646067 4641 log.cc:824] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:43.649346 4641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732: No bootstrap required, opened a new log
I20260619 18:58:43.659426 4641 raft_consensus.cc:348] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813a8b64ea04f57965852fecbf55732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36101 } }
I20260619 18:58:43.659946 4641 raft_consensus.cc:374] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:43.660085 4641 raft_consensus.cc:729] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e813a8b64ea04f57965852fecbf55732, State: Initialized, Role: FOLLOWER
I20260619 18:58:43.660713 4641 consensus_queue.cc:260] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813a8b64ea04f57965852fecbf55732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36101 } }
I20260619 18:58:43.660964 4641 raft_consensus.cc:388] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:43.661072 4641 raft_consensus.cc:482] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:43.661303 4641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:43.664319 4641 raft_consensus.cc:504] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813a8b64ea04f57965852fecbf55732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36101 } }
I20260619 18:58:43.665094 4641 leader_election.cc:302] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e813a8b64ea04f57965852fecbf55732; no voters:
I20260619 18:58:43.665843 4641 leader_election.cc:288] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:43.665947 4651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:43.666785 4651 raft_consensus.cc:686] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [term 1 LEADER]: Becoming Leader. State: Replica: e813a8b64ea04f57965852fecbf55732, State: Running, Role: LEADER
I20260619 18:58:43.667527 4651 consensus_queue.cc:237] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813a8b64ea04f57965852fecbf55732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36101 } }
I20260619 18:58:43.668545 4641 sys_catalog.cc:564] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:43.671207 4652 sys_catalog.cc:455] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e813a8b64ea04f57965852fecbf55732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813a8b64ea04f57965852fecbf55732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36101 } } }
I20260619 18:58:43.671866 4652 sys_catalog.cc:458] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:43.672855 4653 sys_catalog.cc:455] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e813a8b64ea04f57965852fecbf55732. Latest consensus state: current_term: 1 leader_uuid: "e813a8b64ea04f57965852fecbf55732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813a8b64ea04f57965852fecbf55732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36101 } } }
I20260619 18:58:43.673278 4653 sys_catalog.cc:458] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:43.674436 4658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:43.680811 4658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:43.692999 4658 catalog_manager.cc:1347] Generated new cluster ID: ca3543935cba4f81b9cf917c0f78d5d6
I20260619 18:58:43.693172 4658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:43.719463 4658 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:43.720110 4658 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:43.744432 4658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e813a8b64ea04f57965852fecbf55732: Generated new TSK 0
I20260619 18:58:43.745745 4658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:43.761651 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4574
W20260619 18:58:43.763123 4636 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:36037
W20260619 18:58:43.764102 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4574: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:58:43.764314 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 4574
2026-06-19T18:58:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2030 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:43 dist-test-slave-dm3f krb5kdc[4672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:43 dist-test-slave-dm3f krb5kdc[4672](info): set up 2 sockets
Jun 19 18:58:43 dist-test-slave-dm3f krb5kdc[4672](info): commencing operation
krb5kdc: starting...
W20260619 18:58:44.875463 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:45 dist-test-slave-dm3f krb5kdc[4672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:45.154839 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:45.526306 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:45.526643 4685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:45.526850 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:45.534859 4685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:45.534991 4685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:45.535120 4685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:45.535226 4685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:45.535310 4685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:45.546240 4685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:45.547804 4685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:45.549289 4685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:45.562816 4690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:45.562875 4691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:45.562904 4693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:45.564146 4685 server_base.cc:1034] running on GCE node
I20260619 18:58:45.565207 4685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:45.567081 4685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:45.568384 4685 hybrid_clock.cc:648] HybridClock initialized: now 1781895525568343 us; error 65 us; skew 500 ppm
I20260619 18:58:45.568874 4685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:45 dist-test-slave-dm3f krb5kdc[4672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:45.577132 4685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:45.579052 4685 webserver.cc:460] Webserver started at http://127.0.0.1:37477/ using document root <none> and password file <none>
I20260619 18:58:45.579839 4685 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:45.579967 4685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:45.580250 4685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:45.583703 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "0f4ffc79ac794690a7426a0f4ceef7cc"
format_stamp: "Formatted at 2026-06-19 18:58:45 on dist-test-slave-dm3f"
server_key: "ded3912e49c63aecfd3e52396ec002fe"
server_key_iv: "0acd85123407cb1181ad2aa675834d80"
server_key_version: "encryptionkey@0"
I20260619 18:58:45.584656 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "0f4ffc79ac794690a7426a0f4ceef7cc"
format_stamp: "Formatted at 2026-06-19 18:58:45 on dist-test-slave-dm3f"
server_key: "ded3912e49c63aecfd3e52396ec002fe"
server_key_iv: "0acd85123407cb1181ad2aa675834d80"
server_key_version: "encryptionkey@0"
I20260619 18:58:45.591116 4685 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260619 18:58:45.595672 4700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:45.597442 4685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:58:45.597724 4685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "0f4ffc79ac794690a7426a0f4ceef7cc"
format_stamp: "Formatted at 2026-06-19 18:58:45 on dist-test-slave-dm3f"
server_key: "ded3912e49c63aecfd3e52396ec002fe"
server_key_iv: "0acd85123407cb1181ad2aa675834d80"
server_key_version: "encryptionkey@0"
I20260619 18:58:45.597946 4685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:45.613884 4685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:45.619174 4685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:45.619521 4685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:45.641108 4685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37127
I20260619 18:58:45.641139 4751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37127 every 8 connection(s)
I20260619 18:58:45.642841 4685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:45.646278 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 4685
I20260619 18:58:45.646512 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:45.647069 15724 external_mini_cluster.cc:1428] Setting key f4f9bb0463ec10c6d714781344ea28d4
I20260619 18:58:45.647804 4752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:58:45 dist-test-slave-dm3f krb5kdc[4672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:58:45.665149 4752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc: Bootstrap starting.
I20260619 18:58:45.669399 4752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:45.670863 4752 log.cc:824] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:45.674413 4752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc: No bootstrap required, opened a new log
I20260619 18:58:45.675182 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:45.649106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:43465 (local address 127.0.0.1:37127)
0619 18:58:45.650049 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:45.650063 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:45.650112 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:58:45.651928 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:45.651960 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:45.652529 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:45.652961 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:45.655055 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:45.656655 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:45.658669 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:45.659739 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:45.660009 (+ 270us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:45.665855 (+ 5846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:45.665865 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:45.665885 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:45.665943 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:45.669779 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:45.670923 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:45.670931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:45.670940 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:45.671207 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:45.671774 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:45.671779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:45.671781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:45.672451 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:45.672862 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:45.673469 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 18:58:45.673855 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":435,"threads_started":1}
I20260619 18:58:45.680259 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:45.684048 4752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4ffc79ac794690a7426a0f4ceef7cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260619 18:58:45.684542 4752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:45.684677 4752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4ffc79ac794690a7426a0f4ceef7cc, State: Initialized, Role: FOLLOWER
I20260619 18:58:45.685508 4752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4ffc79ac794690a7426a0f4ceef7cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260619 18:58:45.685815 4752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:45.686008 4752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:45.686231 4752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:45.689380 4752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4ffc79ac794690a7426a0f4ceef7cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260619 18:58:45.690116 4752 leader_election.cc:302] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f4ffc79ac794690a7426a0f4ceef7cc; no voters:
I20260619 18:58:45.690912 4752 leader_election.cc:288] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:45.691056 4762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:45.693233 4762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [term 1 LEADER]: Becoming Leader. State: Replica: 0f4ffc79ac794690a7426a0f4ceef7cc, State: Running, Role: LEADER
I20260619 18:58:45.694043 4752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:45.693969 4762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4ffc79ac794690a7426a0f4ceef7cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260619 18:58:45.697726 4763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f4ffc79ac794690a7426a0f4ceef7cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4ffc79ac794690a7426a0f4ceef7cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } } }
I20260619 18:58:45.698750 4764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f4ffc79ac794690a7426a0f4ceef7cc. Latest consensus state: current_term: 1 leader_uuid: "0f4ffc79ac794690a7426a0f4ceef7cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4ffc79ac794690a7426a0f4ceef7cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } } }
I20260619 18:58:45.699422 4764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:45.700011 4763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:45.704974 4770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:45.706440 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:45.685136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:42215 (local address 127.0.0.1:37127)
0619 18:58:45.685353 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:45.685360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:45.685893 (+ 533us) server_negotiation.cc:407] Connection header received
0619 18:58:45.686105 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:45.686110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:45.686212 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:45.686342 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:45.688164 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:45.689326 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:45.691003 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:45.691927 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:45.692089 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:45.695588 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:45.695601 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:45.695606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:45.695667 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:45.700621 (+ 4954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:45.702243 (+ 1622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:45.702252 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:45.702308 (+ 56us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:45.702738 (+ 430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:45.703567 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:45.703575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:45.703581 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:45.704132 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:45.704541 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:45.705778 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0619 18:58:45.706028 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:58:45.711993 4770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:45.724104 4770 catalog_manager.cc:1347] Generated new cluster ID: 6d37912ab24742eba5d1b24e5f051fa5
I20260619 18:58:45.724263 4770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:45.742815 4770 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:45.744833 4770 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:45.762912 4770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f4ffc79ac794690a7426a0f4ceef7cc: Generated new TSK 0
I20260619 18:58:45.764143 4770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:45.780982 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4685
I20260619 18:58:45.782418 4747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:45.880184 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 4685
W20260619 18:58:45.898912 4670 connection.cc:535] client connection to 127.0.0.1:37127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37127): Connection reset by peer (error 104)
2026-06-19T18:58:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2118 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:45 dist-test-slave-dm3f krb5kdc[4784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:45 dist-test-slave-dm3f krb5kdc[4784](info): set up 2 sockets
Jun 19 18:58:45 dist-test-slave-dm3f krb5kdc[4784](info): commencing operation
krb5kdc: starting...
W20260619 18:58:47.007720 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:47 dist-test-slave-dm3f krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:47.293601 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:47.668031 4797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:47.668440 4797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:47.668610 4797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:47.677219 4797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:47.677426 4797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:47.677528 4797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:47.677640 4797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:47.677719 4797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:47.688860 4797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:47.690455 4797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:47.691999 4797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:47.706085 4802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:47.706068 4805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:47.706022 4803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:47.707054 4797 server_base.cc:1034] running on GCE node
I20260619 18:58:47.708037 4797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:47.709796 4797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:47.711097 4797 hybrid_clock.cc:648] HybridClock initialized: now 1781895527711041 us; error 57 us; skew 500 ppm
I20260619 18:58:47.711557 4797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:47 dist-test-slave-dm3f krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:47.716010 4797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:47.718175 4797 webserver.cc:460] Webserver started at http://127.0.0.1:33553/ using document root <none> and password file <none>
I20260619 18:58:47.719134 4797 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:47.719324 4797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:47.719681 4797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:47.723299 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "a2d5d7ec6ae449cbbe6fe5bcee31a260"
format_stamp: "Formatted at 2026-06-19 18:58:47 on dist-test-slave-dm3f"
server_key: "14212851ca59b9522e1fc5430dc496bb"
server_key_iv: "4ed5b0a1a8a356911b2336982d40a364"
server_key_version: "encryptionkey@0"
I20260619 18:58:47.724260 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "a2d5d7ec6ae449cbbe6fe5bcee31a260"
format_stamp: "Formatted at 2026-06-19 18:58:47 on dist-test-slave-dm3f"
server_key: "14212851ca59b9522e1fc5430dc496bb"
server_key_iv: "4ed5b0a1a8a356911b2336982d40a364"
server_key_version: "encryptionkey@0"
I20260619 18:58:47.731066 4797 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260619 18:58:47.735641 4812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:47.737564 4797 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:58:47.737843 4797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "a2d5d7ec6ae449cbbe6fe5bcee31a260"
format_stamp: "Formatted at 2026-06-19 18:58:47 on dist-test-slave-dm3f"
server_key: "14212851ca59b9522e1fc5430dc496bb"
server_key_iv: "4ed5b0a1a8a356911b2336982d40a364"
server_key_version: "encryptionkey@0"
I20260619 18:58:47.738108 4797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:47.757836 4797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:47.763017 4797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:47.763367 4797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:47.785035 4797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34687
I20260619 18:58:47.785059 4863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34687 every 8 connection(s)
I20260619 18:58:47.786692 4797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:47.791107 4864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:47.792876 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 4797
I20260619 18:58:47.793074 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:47.793718 15724 external_mini_cluster.cc:1428] Setting key 3e0b027be07393780435ef6927eebc91
I20260619 18:58:47.808313 4864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260: Bootstrap starting.
I20260619 18:58:47.810580 4867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:47.795906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:52377 (local address 127.0.0.1:34687)
0619 18:58:47.796968 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:47.796981 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:47.797038 (+ 57us) server_negotiation.cc:407] Connection header received
0619 18:58:47.799020 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:47.799060 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:47.799663 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:47.800090 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:47.800869 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:47.802907 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:47.805072 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:47.805966 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:47.806178 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:47.807722 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:47.807732 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:47.807752 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:47.808469 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:47.808593 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:47.808629 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:47.808783 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:47.809197 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 18:58:47.809427 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":501,"threads_started":1}
I20260619 18:58:47.812332 4864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:47.813659 4864 log.cc:824] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:47.815696 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:47.817947 4864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260: No bootstrap required, opened a new log
I20260619 18:58:47.829409 4867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:47.821524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:58909 (local address 127.0.0.1:34687)
0619 18:58:47.821751 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:47.821759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:47.822295 (+ 536us) server_negotiation.cc:407] Connection header received
0619 18:58:47.822477 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:47.822484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:47.822593 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:47.822767 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:47.823375 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:47.824646 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:47.826203 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:47.826986 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:47.827167 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:47.828503 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:47.828509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:47.828514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:47.828541 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:47.828626 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:47.828650 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:47.828796 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:47.828982 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 18:58:47.829048 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:58:47.829332 4864 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d5d7ec6ae449cbbe6fe5bcee31a260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } }
I20260619 18:58:47.829980 4864 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:47.830094 4864 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d5d7ec6ae449cbbe6fe5bcee31a260, State: Initialized, Role: FOLLOWER
I20260619 18:58:47.830941 4864 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d5d7ec6ae449cbbe6fe5bcee31a260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } }
I20260619 18:58:47.831282 4864 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:47.831496 4864 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:47.831700 4864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:47.834223 4864 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d5d7ec6ae449cbbe6fe5bcee31a260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } }
I20260619 18:58:47.834993 4864 leader_election.cc:302] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2d5d7ec6ae449cbbe6fe5bcee31a260; no voters:
I20260619 18:58:47.835737 4864 leader_election.cc:288] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:47.835896 4874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:47.837761 4874 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [term 1 LEADER]: Becoming Leader. State: Replica: a2d5d7ec6ae449cbbe6fe5bcee31a260, State: Running, Role: LEADER
I20260619 18:58:47.838644 4864 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:47.838531 4874 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d5d7ec6ae449cbbe6fe5bcee31a260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } }
I20260619 18:58:47.841940 4876 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d5d7ec6ae449cbbe6fe5bcee31a260. Latest consensus state: current_term: 1 leader_uuid: "a2d5d7ec6ae449cbbe6fe5bcee31a260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d5d7ec6ae449cbbe6fe5bcee31a260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } } }
I20260619 18:58:47.842526 4876 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:47.843334 4875 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2d5d7ec6ae449cbbe6fe5bcee31a260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d5d7ec6ae449cbbe6fe5bcee31a260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } } }
I20260619 18:58:47.843747 4875 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:47.847536 4883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:47.853835 4883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:47.865372 4883 catalog_manager.cc:1347] Generated new cluster ID: 85a688b4b68d4aceba473fc99ae91322
I20260619 18:58:47.865548 4883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:47.889925 4883 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:47.892030 4883 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:47.925043 4883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2d5d7ec6ae449cbbe6fe5bcee31a260: Generated new TSK 0
I20260619 18:58:47.926296 4883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:47.947098 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4797
W20260619 18:58:47.949273 4859 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:52377
W20260619 18:58:47.950243 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4797: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:58:47.950435 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 4797
W20260619 18:58:47.968658 4782 connection.cc:535] client connection to 127.0.0.1:34687 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34687): Connection reset by peer (error 104)
2026-06-19T18:58:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2069 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:48 dist-test-slave-dm3f krb5kdc[4895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:48 dist-test-slave-dm3f krb5kdc[4895](info): set up 2 sockets
Jun 19 18:58:48 dist-test-slave-dm3f krb5kdc[4895](info): commencing operation
krb5kdc: starting...
W20260619 18:58:49.095323 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:49 dist-test-slave-dm3f krb5kdc[4895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:58:49.374377 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:49.752218 4908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:49.752565 4908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:49.752743 4908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:49.760746 4908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:49.760897 4908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:49.760991 4908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:49.761092 4908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:49.761200 4908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:49.771955 4908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:49.773478 4908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:49.775010 4908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:49.788808 4913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:49.788864 4914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:49.788888 4916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:49.790688 4908 server_base.cc:1034] running on GCE node
I20260619 18:58:49.791692 4908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:49.793506 4908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:49.794878 4908 hybrid_clock.cc:648] HybridClock initialized: now 1781895529794817 us; error 46 us; skew 500 ppm
I20260619 18:58:49.795380 4908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:49 dist-test-slave-dm3f krb5kdc[4895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:49.799300 4908 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:58:49.801327 4908 webserver.cc:460] Webserver started at http://127.0.0.1:36989/ using document root <none> and password file <none>
I20260619 18:58:49.802229 4908 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:49.802430 4908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:49.802876 4908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:49.806406 4908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "b00afe2261b04ad099ff876545fa0a7b"
format_stamp: "Formatted at 2026-06-19 18:58:49 on dist-test-slave-dm3f"
server_key: "44d676b2f3eee7fb5a34768fa75def53"
server_key_iv: "9b70bd710251ed77a77095e61f8e79eb"
server_key_version: "encryptionkey@0"
I20260619 18:58:49.807511 4908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "b00afe2261b04ad099ff876545fa0a7b"
format_stamp: "Formatted at 2026-06-19 18:58:49 on dist-test-slave-dm3f"
server_key: "44d676b2f3eee7fb5a34768fa75def53"
server_key_iv: "9b70bd710251ed77a77095e61f8e79eb"
server_key_version: "encryptionkey@0"
I20260619 18:58:49.813935 4908 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:58:49.818622 4923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:49.820607 4908 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:49.820901 4908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "b00afe2261b04ad099ff876545fa0a7b"
format_stamp: "Formatted at 2026-06-19 18:58:49 on dist-test-slave-dm3f"
server_key: "44d676b2f3eee7fb5a34768fa75def53"
server_key_iv: "9b70bd710251ed77a77095e61f8e79eb"
server_key_version: "encryptionkey@0"
I20260619 18:58:49.821170 4908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:49.828118 4908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:49.833151 4908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:49.833528 4908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:49.855216 4908 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33159
I20260619 18:58:49.855268 4974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33159 every 8 connection(s)
I20260619 18:58:49.856897 4908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:49.857344 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 4908
I20260619 18:58:49.857523 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:49.857986 15724 external_mini_cluster.cc:1428] Setting key 6efc5c98d9c4cdd1701e5ca58d77c579
I20260619 18:58:49.862579 4975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:49.868219 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:49.860191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:44901 (local address 127.0.0.1:33159)
0619 18:58:49.861133 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:49.861150 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:49.861213 (+ 63us) server_negotiation.cc:407] Connection header received
0619 18:58:49.863666 (+ 2453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:49.863701 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:49.864219 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:49.864731 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:49.865866 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:49.865875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:49.865895 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:49.865916 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:49.866038 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:49.866076 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:49.866351 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:49.866511 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 18:58:49.866811 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":376,"threads_started":1}
I20260619 18:58:49.873271 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:49.877851 4975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b: Bootstrap starting.
I20260619 18:58:49.881242 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:49.878249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:46609 (local address 127.0.0.1:33159)
0619 18:58:49.878475 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:49.878483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:49.878940 (+ 457us) server_negotiation.cc:407] Connection header received
0619 18:58:49.879179 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:49.879183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:49.879244 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:49.879358 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:49.880489 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:49.880494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:49.880497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:49.880516 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:49.880562 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:49.880579 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:49.880716 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:49.880854 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 18:58:49.880933 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:58:49.882175 4975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:49.883642 4975 log.cc:824] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:49.887408 4975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b: No bootstrap required, opened a new log
I20260619 18:58:49.896656 4975 raft_consensus.cc:348] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00afe2261b04ad099ff876545fa0a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33159 } }
I20260619 18:58:49.897114 4975 raft_consensus.cc:374] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:49.897269 4975 raft_consensus.cc:729] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00afe2261b04ad099ff876545fa0a7b, State: Initialized, Role: FOLLOWER
I20260619 18:58:49.897958 4975 consensus_queue.cc:260] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00afe2261b04ad099ff876545fa0a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33159 } }
I20260619 18:58:49.898257 4975 raft_consensus.cc:388] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:49.898435 4975 raft_consensus.cc:482] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:49.898649 4975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:49.901232 4975 raft_consensus.cc:504] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00afe2261b04ad099ff876545fa0a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33159 } }
I20260619 18:58:49.901974 4975 leader_election.cc:302] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b00afe2261b04ad099ff876545fa0a7b; no voters:
I20260619 18:58:49.902889 4975 leader_election.cc:288] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:49.903025 4985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:49.903815 4985 raft_consensus.cc:686] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [term 1 LEADER]: Becoming Leader. State: Replica: b00afe2261b04ad099ff876545fa0a7b, State: Running, Role: LEADER
I20260619 18:58:49.904557 4985 consensus_queue.cc:237] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00afe2261b04ad099ff876545fa0a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33159 } }
I20260619 18:58:49.905443 4975 sys_catalog.cc:564] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:49.908120 4987 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b00afe2261b04ad099ff876545fa0a7b. Latest consensus state: current_term: 1 leader_uuid: "b00afe2261b04ad099ff876545fa0a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00afe2261b04ad099ff876545fa0a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33159 } } }
I20260619 18:58:49.908118 4986 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b00afe2261b04ad099ff876545fa0a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00afe2261b04ad099ff876545fa0a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33159 } } }
I20260619 18:58:49.908828 4987 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:49.908902 4986 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:49.909989 4992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:49.916357 4992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:49.927915 4992 catalog_manager.cc:1347] Generated new cluster ID: 2084326825114ca7a195af7fbf74a27e
I20260619 18:58:49.928077 4992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:49.943997 4992 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:49.944587 4992 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:49.954695 4992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b00afe2261b04ad099ff876545fa0a7b: Generated new TSK 0
I20260619 18:58:49.956032 4992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:49.962884 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4908
W20260619 18:58:49.964283 4970 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:44901
W20260619 18:58:49.965291 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 4908: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:58:49.965512 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 4908
2026-06-19T18:58:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2015 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:50 dist-test-slave-dm3f krb5kdc[5006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:50 dist-test-slave-dm3f krb5kdc[5006](info): set up 2 sockets
Jun 19 18:58:50 dist-test-slave-dm3f krb5kdc[5006](info): commencing operation
krb5kdc: starting...
W20260619 18:58:51.095516 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:51 dist-test-slave-dm3f krb5kdc[5006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:58:51.386844 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:42161
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:42161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:51.794108 5019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:51.794456 5019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:51.794633 5019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:51.802667 5019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:51.802920 5019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:51.803036 5019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:51.803146 5019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:51.803243 5019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:51.814986 5019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:42161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:42161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:51.816534 5019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:51.818089 5019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:51.831679 5025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:51.831656 5024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:51.831637 5027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:51.833043 5019 server_base.cc:1034] running on GCE node
I20260619 18:58:51.834082 5019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:51.835978 5019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:51.837286 5019 hybrid_clock.cc:648] HybridClock initialized: now 1781895531837184 us; error 70 us; skew 500 ppm
I20260619 18:58:51.837759 5019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:51 dist-test-slave-dm3f krb5kdc[5006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895531, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:51.842028 5019 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:58:51.844103 5019 webserver.cc:460] Webserver started at http://127.15.91.62:40537/ using document root <none> and password file <none>
I20260619 18:58:51.844971 5019 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:51.845165 5019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:51.845536 5019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:51.849098 5019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "6e91be952611436ebfe1d7848b927a7a"
format_stamp: "Formatted at 2026-06-19 18:58:51 on dist-test-slave-dm3f"
server_key: "f523ee0dde369647118d60fedf779b80"
server_key_iv: "f8f80a55937ee21eb264e594d546cebe"
server_key_version: "encryptionkey@0"
I20260619 18:58:51.850131 5019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "6e91be952611436ebfe1d7848b927a7a"
format_stamp: "Formatted at 2026-06-19 18:58:51 on dist-test-slave-dm3f"
server_key: "f523ee0dde369647118d60fedf779b80"
server_key_iv: "f8f80a55937ee21eb264e594d546cebe"
server_key_version: "encryptionkey@0"
I20260619 18:58:51.857105 5019 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260619 18:58:51.861761 5034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:51.863823 5019 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:58:51.864125 5019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "6e91be952611436ebfe1d7848b927a7a"
format_stamp: "Formatted at 2026-06-19 18:58:51 on dist-test-slave-dm3f"
server_key: "f523ee0dde369647118d60fedf779b80"
server_key_iv: "f8f80a55937ee21eb264e594d546cebe"
server_key_version: "encryptionkey@0"
I20260619 18:58:51.864408 5019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:51.881202 5019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:51.886216 5019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:51.886592 5019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:51.909762 5019 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:42161
I20260619 18:58:51.909874 5085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:42161 every 8 connection(s)
I20260619 18:58:51.911610 5019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:51.916043 5086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:51.919960 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 5019
I20260619 18:58:51.920181 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:51.920629 15724 external_mini_cluster.cc:1428] Setting key df09c427f41cbc6d3ba74ad4f55db1aa
I20260619 18:58:51.931394 5086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a: Bootstrap starting.
I20260619 18:58:51.935791 5086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:51.937297 5086 log.cc:824] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:58:51 dist-test-slave-dm3f krb5kdc[5006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:51.941756 5086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a: No bootstrap required, opened a new log
I20260619 18:58:51.953931 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:51.922934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52671 (local address 127.15.91.62:42161)
0619 18:58:51.923912 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:51.923930 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:51.923992 (+ 62us) server_negotiation.cc:407] Connection header received
0619 18:58:51.926482 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:51.926536 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:51.927321 (+ 785us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:51.927930 (+ 609us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:51.930013 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:51.932175 (+ 2162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:51.933763 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:51.935027 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:51.935280 (+ 253us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:51.941570 (+ 6290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:51.941585 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:51.941617 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:51.941707 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:51.947555 (+ 5848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:51.948838 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:51.948851 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:51.948867 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:51.949179 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:51.949779 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:51.949790 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:51.949794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:51.950776 (+ 982us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:51.951164 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:51.951727 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 18:58:51.952193 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":420,"threads_started":1}
I20260619 18:58:51.954831 5086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e91be952611436ebfe1d7848b927a7a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42161 } }
I20260619 18:58:51.955376 5086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:51.955505 5086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e91be952611436ebfe1d7848b927a7a, State: Initialized, Role: FOLLOWER
I20260619 18:58:51.956197 5086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e91be952611436ebfe1d7848b927a7a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42161 } }
I20260619 18:58:51.956511 5086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:51.956696 5086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:51.956898 5086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:51.959870 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:51.960206 5086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e91be952611436ebfe1d7848b927a7a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42161 } }
I20260619 18:58:51.960870 5086 leader_election.cc:302] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e91be952611436ebfe1d7848b927a7a; no voters:
I20260619 18:58:51.961581 5086 leader_election.cc:288] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:51.961817 5091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:51.963263 5091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [term 1 LEADER]: Becoming Leader. State: Replica: 6e91be952611436ebfe1d7848b927a7a, State: Running, Role: LEADER
I20260619 18:58:51.964293 5086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:51.964193 5091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e91be952611436ebfe1d7848b927a7a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42161 } }
I20260619 18:58:51.968863 5092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e91be952611436ebfe1d7848b927a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e91be952611436ebfe1d7848b927a7a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42161 } } }
I20260619 18:58:51.969452 5092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:51.970112 5093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e91be952611436ebfe1d7848b927a7a. Latest consensus state: current_term: 1 leader_uuid: "6e91be952611436ebfe1d7848b927a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e91be952611436ebfe1d7848b927a7a" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 42161 } } }
I20260619 18:58:51.970512 5093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:51.979089 5099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:51.986121 5099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:51.995309 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:51.972539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54845 (local address 127.15.91.62:42161)
0619 18:58:51.972762 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:51.972772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:51.973265 (+ 493us) server_negotiation.cc:407] Connection header received
0619 18:58:51.973831 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:51.973839 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:51.973940 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:51.974043 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:51.976432 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:51.977839 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:51.980066 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:51.980810 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:51.980995 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:51.985177 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:51.985186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:51.985191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:51.985258 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:51.989850 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:51.991290 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:51.991301 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:51.991305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:51.991510 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:51.992195 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:51.992201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:51.992205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:51.992739 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:51.993069 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:51.994656 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0619 18:58:51.994923 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:58:52.009624 5099 catalog_manager.cc:1347] Generated new cluster ID: 0c0db8d234c74b7b906e1b2bf88a54db
I20260619 18:58:52.010051 5099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:52.038638 5099 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:52.041016 5099 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:52.050151 5099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e91be952611436ebfe1d7848b927a7a: Generated new TSK 0
I20260619 18:58:52.051294 5099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:52.061339 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5019
I20260619 18:58:52.062867 5081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:52.164641 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 5019
W20260619 18:58:52.183331 5004 connection.cc:535] client connection to 127.15.91.62:42161 recv error: Network error: failed to read from TLS socket (remote: 127.15.91.62:42161): Connection reset by peer (error 104)
2026-06-19T18:58:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:52 dist-test-slave-dm3f krb5kdc[5118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:52 dist-test-slave-dm3f krb5kdc[5118](info): set up 2 sockets
Jun 19 18:58:52 dist-test-slave-dm3f krb5kdc[5118](info): commencing operation
krb5kdc: starting...
W20260619 18:58:53.295842 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:53 dist-test-slave-dm3f krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:58:53.598475 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38811
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:38811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:53.976986 5131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:53.977362 5131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:53.977568 5131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:53.985570 5131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:53.985730 5131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:53.985827 5131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:53.985950 5131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:53.986042 5131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:53.997509 5131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:38811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:38811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:53.999200 5131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:54.000813 5131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:54.017079 5137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:54.017370 5139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:54.017266 5136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:54.019263 5131 server_base.cc:1034] running on GCE node
I20260619 18:58:54.020359 5131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:54.022225 5131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:54.023643 5131 hybrid_clock.cc:648] HybridClock initialized: now 1781895534023553 us; error 61 us; skew 500 ppm
I20260619 18:58:54.024330 5131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:54 dist-test-slave-dm3f krb5kdc[5118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895534, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:54.029100 5131 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:58:54.031338 5131 webserver.cc:460] Webserver started at http://127.15.91.62:34523/ using document root <none> and password file <none>
I20260619 18:58:54.032245 5131 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:54.032442 5131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:54.032809 5131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:54.037256 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "d6b807cade5546cfb1904dcc0c628c31"
format_stamp: "Formatted at 2026-06-19 18:58:54 on dist-test-slave-dm3f"
server_key: "f08d971484db7bfbb636b4248290c04b"
server_key_iv: "9de868a76ebf3f81604dce4f6c715200"
server_key_version: "encryptionkey@0"
I20260619 18:58:54.038357 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "d6b807cade5546cfb1904dcc0c628c31"
format_stamp: "Formatted at 2026-06-19 18:58:54 on dist-test-slave-dm3f"
server_key: "f08d971484db7bfbb636b4248290c04b"
server_key_iv: "9de868a76ebf3f81604dce4f6c715200"
server_key_version: "encryptionkey@0"
I20260619 18:58:54.045805 5131 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 18:58:54.051874 5146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:54.054286 5131 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:58:54.054595 5131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "d6b807cade5546cfb1904dcc0c628c31"
format_stamp: "Formatted at 2026-06-19 18:58:54 on dist-test-slave-dm3f"
server_key: "f08d971484db7bfbb636b4248290c04b"
server_key_iv: "9de868a76ebf3f81604dce4f6c715200"
server_key_version: "encryptionkey@0"
I20260619 18:58:54.054999 5131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:54.095772 5131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:54.101253 5131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:54.101624 5131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:54.124634 5131 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:38811
I20260619 18:58:54.124657 5197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:38811 every 8 connection(s)
I20260619 18:58:54.126318 5131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:54.131002 5198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:54.132091 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 5131
I20260619 18:58:54.132293 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:54.132843 15724 external_mini_cluster.cc:1428] Setting key daa7bd3eaef151d19c1c9e0ea8baea61
I20260619 18:58:54.147796 5198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31: Bootstrap starting.
I20260619 18:58:54.150307 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:54.135175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45005 (local address 127.15.91.62:38811)
0619 18:58:54.136213 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:54.136228 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:54.136280 (+ 52us) server_negotiation.cc:407] Connection header received
0619 18:58:54.138358 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:54.138397 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:54.139044 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:54.139478 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:54.140272 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:54.141681 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:54.143911 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:54.144882 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:54.145090 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:54.147100 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:54.147117 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:54.147146 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:54.148193 (+ 1047us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:54.148308 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:54.148349 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:54.148534 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:54.148936 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 18:58:54.149112 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":476,"threads_started":1}
I20260619 18:58:54.151945 5198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:54.153322 5198 log.cc:824] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:54.155732 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:54.158195 5198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31: No bootstrap required, opened a new log
I20260619 18:58:54.170446 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:54.161210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60523 (local address 127.15.91.62:38811)
0619 18:58:54.161512 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:54.161520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:54.162021 (+ 501us) server_negotiation.cc:407] Connection header received
0619 18:58:54.162241 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:54.162246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:54.162334 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:54.162466 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:54.163313 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:54.164198 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:54.166630 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:54.167305 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:54.167480 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:54.169226 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:54.169237 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:54.169242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:54.169278 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:54.169358 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:54.169381 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:54.169540 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:54.169990 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 18:58:54.170075 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 18:58:54.170935 5198 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b807cade5546cfb1904dcc0c628c31" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38811 } }
I20260619 18:58:54.171473 5198 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:54.171623 5198 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b807cade5546cfb1904dcc0c628c31, State: Initialized, Role: FOLLOWER
I20260619 18:58:54.172293 5198 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b807cade5546cfb1904dcc0c628c31" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38811 } }
I20260619 18:58:54.172592 5198 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:54.172756 5198 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:54.172955 5198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:54.175644 5198 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b807cade5546cfb1904dcc0c628c31" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38811 } }
I20260619 18:58:54.176390 5198 leader_election.cc:302] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6b807cade5546cfb1904dcc0c628c31; no voters:
I20260619 18:58:54.177429 5198 leader_election.cc:288] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:54.177626 5208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:54.178476 5208 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [term 1 LEADER]: Becoming Leader. State: Replica: d6b807cade5546cfb1904dcc0c628c31, State: Running, Role: LEADER
I20260619 18:58:54.179339 5208 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b807cade5546cfb1904dcc0c628c31" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38811 } }
I20260619 18:58:54.180270 5198 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:54.184633 5209 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6b807cade5546cfb1904dcc0c628c31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b807cade5546cfb1904dcc0c628c31" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38811 } } }
I20260619 18:58:54.185194 5209 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:54.187013 5210 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6b807cade5546cfb1904dcc0c628c31. Latest consensus state: current_term: 1 leader_uuid: "d6b807cade5546cfb1904dcc0c628c31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b807cade5546cfb1904dcc0c628c31" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 38811 } } }
I20260619 18:58:54.187578 5210 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:54.188599 5217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:54.194744 5217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:54.207073 5217 catalog_manager.cc:1347] Generated new cluster ID: a6f01c1ff476483ea696ac896fcae735
I20260619 18:58:54.207249 5217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:54.223152 5217 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:54.224745 5217 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:54.232540 5217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6b807cade5546cfb1904dcc0c628c31: Generated new TSK 0
I20260619 18:58:54.233642 5217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:54.246513 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5131
W20260619 18:58:54.248642 5193 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45005
W20260619 18:58:54.249850 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5131: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:58:54.250090 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 5131
W20260619 18:58:54.266004 5116 connection.cc:535] client connection to 127.15.91.62:38811 recv error: Network error: failed to read from TLS socket (remote: 127.15.91.62:38811): Connection reset by peer (error 104)
2026-06-19T18:58:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2082 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:54 dist-test-slave-dm3f krb5kdc[5229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:54 dist-test-slave-dm3f krb5kdc[5229](info): set up 2 sockets
Jun 19 18:58:54 dist-test-slave-dm3f krb5kdc[5229](info): commencing operation
krb5kdc: starting...
W20260619 18:58:55.388055 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:55 dist-test-slave-dm3f krb5kdc[5229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:58:55.684631 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:41459
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:41459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:56.057587 5242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:56.057927 5242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:56.058027 5242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:56.065783 5242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:56.065927 5242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:56.065975 5242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:56.066041 5242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:56.066090 5242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:56.076644 5242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:41459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:41459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:56.078023 5242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:56.079448 5242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:56.093918 5248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:56.094024 5247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:56.094074 5250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:56.095273 5242 server_base.cc:1034] running on GCE node
I20260619 18:58:56.096159 5242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:56.097869 5242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:56.099188 5242 hybrid_clock.cc:648] HybridClock initialized: now 1781895536099106 us; error 68 us; skew 500 ppm
I20260619 18:58:56.099679 5242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:56 dist-test-slave-dm3f krb5kdc[5229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895536, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:56.103556 5242 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:58:56.105471 5242 webserver.cc:460] Webserver started at http://127.15.91.62:35247/ using document root <none> and password file <none>
I20260619 18:58:56.106355 5242 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:56.106549 5242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:56.106983 5242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:56.110415 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "e57d8fd58f0d46a3b7af0a2c7db817cb"
format_stamp: "Formatted at 2026-06-19 18:58:56 on dist-test-slave-dm3f"
server_key: "8cc07f1e33da5f2e0c6ed93daa1c58f9"
server_key_iv: "6c2705f33169ae37e5fafeadddd7fb3a"
server_key_version: "encryptionkey@0"
I20260619 18:58:56.111490 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "e57d8fd58f0d46a3b7af0a2c7db817cb"
format_stamp: "Formatted at 2026-06-19 18:58:56 on dist-test-slave-dm3f"
server_key: "8cc07f1e33da5f2e0c6ed93daa1c58f9"
server_key_iv: "6c2705f33169ae37e5fafeadddd7fb3a"
server_key_version: "encryptionkey@0"
I20260619 18:58:56.118479 5242 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:58:56.123404 5257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:56.125562 5242 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:58:56.125882 5242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "e57d8fd58f0d46a3b7af0a2c7db817cb"
format_stamp: "Formatted at 2026-06-19 18:58:56 on dist-test-slave-dm3f"
server_key: "8cc07f1e33da5f2e0c6ed93daa1c58f9"
server_key_iv: "6c2705f33169ae37e5fafeadddd7fb3a"
server_key_version: "encryptionkey@0"
I20260619 18:58:56.126153 5242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:56.133131 5242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:56.138855 5242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:56.139303 5242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:56.161007 5242 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:41459
I20260619 18:58:56.161059 5308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:41459 every 8 connection(s)
I20260619 18:58:56.162767 5242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:56.167483 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 5242
I20260619 18:58:56.167721 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:56.168190 15724 external_mini_cluster.cc:1428] Setting key a6ea553419f075042644f317803672d3
I20260619 18:58:56.168828 5309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:56.179677 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:56.170932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39293 (local address 127.15.91.62:41459)
0619 18:58:56.172140 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:56.172160 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:56.172234 (+ 74us) server_negotiation.cc:407] Connection header received
0619 18:58:56.174773 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:56.174817 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:56.175449 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:56.176057 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:56.177315 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:56.177324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:56.177344 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:56.177368 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:56.177446 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:56.177482 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:56.177762 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:56.177938 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 18:58:56.178193 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":496,"threads_started":1}
I20260619 18:58:56.184535 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:56.185820 5309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb: Bootstrap starting.
I20260619 18:58:56.190186 5309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:56.192066 5309 log.cc:824] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:56.194191 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:56.190480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45045 (local address 127.15.91.62:41459)
0619 18:58:56.190806 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:56.190817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:56.191366 (+ 549us) server_negotiation.cc:407] Connection header received
0619 18:58:56.191598 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:56.191603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:56.191669 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:56.191764 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:56.193062 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:56.193069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:56.193075 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:58:56.193100 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:56.193153 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:58:56.193178 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:56.193358 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:56.193723 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 18:58:56.193811 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 18:58:56.196748 5309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb: No bootstrap required, opened a new log
I20260619 18:58:56.205420 5309 raft_consensus.cc:348] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57d8fd58f0d46a3b7af0a2c7db817cb" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41459 } }
I20260619 18:58:56.205947 5309 raft_consensus.cc:374] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:56.206065 5309 raft_consensus.cc:729] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57d8fd58f0d46a3b7af0a2c7db817cb, State: Initialized, Role: FOLLOWER
I20260619 18:58:56.206907 5309 consensus_queue.cc:260] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57d8fd58f0d46a3b7af0a2c7db817cb" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41459 } }
I20260619 18:58:56.207234 5309 raft_consensus.cc:388] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:56.207408 5309 raft_consensus.cc:482] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:56.207602 5309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:56.210134 5309 raft_consensus.cc:504] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57d8fd58f0d46a3b7af0a2c7db817cb" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41459 } }
I20260619 18:58:56.210888 5309 leader_election.cc:302] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e57d8fd58f0d46a3b7af0a2c7db817cb; no voters:
I20260619 18:58:56.211571 5309 leader_election.cc:288] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:56.211758 5319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:56.212555 5319 raft_consensus.cc:686] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [term 1 LEADER]: Becoming Leader. State: Replica: e57d8fd58f0d46a3b7af0a2c7db817cb, State: Running, Role: LEADER
I20260619 18:58:56.213195 5319 consensus_queue.cc:237] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57d8fd58f0d46a3b7af0a2c7db817cb" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41459 } }
I20260619 18:58:56.214057 5309 sys_catalog.cc:564] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:56.216553 5321 sys_catalog.cc:455] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e57d8fd58f0d46a3b7af0a2c7db817cb. Latest consensus state: current_term: 1 leader_uuid: "e57d8fd58f0d46a3b7af0a2c7db817cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57d8fd58f0d46a3b7af0a2c7db817cb" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41459 } } }
I20260619 18:58:56.217540 5320 sys_catalog.cc:455] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e57d8fd58f0d46a3b7af0a2c7db817cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57d8fd58f0d46a3b7af0a2c7db817cb" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 41459 } } }
I20260619 18:58:56.218066 5321 sys_catalog.cc:458] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:56.218084 5320 sys_catalog.cc:458] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:56.221370 5327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:56.227124 5327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:56.240326 5327 catalog_manager.cc:1347] Generated new cluster ID: 67ba44d4e489492cbc090597dbd30d77
I20260619 18:58:56.240479 5327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:56.261116 5327 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:56.261687 5327 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:56.269835 5327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e57d8fd58f0d46a3b7af0a2c7db817cb: Generated new TSK 0
I20260619 18:58:56.270923 5327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:56.284584 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5242
W20260619 18:58:56.286423 5304 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39293
W20260619 18:58:56.287371 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5242: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:58:56.287580 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 5242
2026-06-19T18:58:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:56 dist-test-slave-dm3f krb5kdc[5340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:56 dist-test-slave-dm3f krb5kdc[5340](info): set up 2 sockets
Jun 19 18:58:56 dist-test-slave-dm3f krb5kdc[5340](info): commencing operation
krb5kdc: starting...
W20260619 18:58:57.431658 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:57 dist-test-slave-dm3f krb5kdc[5340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:58:57.710853 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:33981
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:33981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:58:58.082342 5353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:58:58.082777 5353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:58:58.082964 5353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:58:58.091020 5353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:58:58.091223 5353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:58:58.091351 5353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:58:58.091468 5353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:58:58.091554 5353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:58:58.104136 5353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:33981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:33981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:58:58.105681 5353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:58:58.107337 5353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:58:58.122699 5358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:58.122531 5361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:58:58.122550 5359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:58:58.123371 5353 server_base.cc:1034] running on GCE node
I20260619 18:58:58.124397 5353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:58:58.126065 5353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:58:58.127328 5353 hybrid_clock.cc:648] HybridClock initialized: now 1781895538127265 us; error 56 us; skew 500 ppm
I20260619 18:58:58.127727 5353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:58:58 dist-test-slave-dm3f krb5kdc[5340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895538, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:58:58.135021 5353 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:58:58.136994 5353 webserver.cc:460] Webserver started at http://127.15.91.62:34337/ using document root <none> and password file <none>
I20260619 18:58:58.137926 5353 fs_manager.cc:359] Metadata directory not provided
I20260619 18:58:58.138126 5353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:58:58.138490 5353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:58:58.142077 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "9c3f3d74f2e54bd4a70595ffef1aad95"
format_stamp: "Formatted at 2026-06-19 18:58:58 on dist-test-slave-dm3f"
server_key: "2f2dd64b586959d92d96a44ee609a9b6"
server_key_iv: "1bad9486849edb973b22523c6423d743"
server_key_version: "encryptionkey@0"
I20260619 18:58:58.143151 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "9c3f3d74f2e54bd4a70595ffef1aad95"
format_stamp: "Formatted at 2026-06-19 18:58:58 on dist-test-slave-dm3f"
server_key: "2f2dd64b586959d92d96a44ee609a9b6"
server_key_iv: "1bad9486849edb973b22523c6423d743"
server_key_version: "encryptionkey@0"
I20260619 18:58:58.149585 5353 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260619 18:58:58.154094 5368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:58:58.156028 5353 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:58:58.156332 5353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "9c3f3d74f2e54bd4a70595ffef1aad95"
format_stamp: "Formatted at 2026-06-19 18:58:58 on dist-test-slave-dm3f"
server_key: "2f2dd64b586959d92d96a44ee609a9b6"
server_key_iv: "1bad9486849edb973b22523c6423d743"
server_key_version: "encryptionkey@0"
I20260619 18:58:58.156601 5353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:58:58.176086 5353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:58:58.181517 5353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:58:58.181908 5353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:58:58.204295 5353 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:33981
I20260619 18:58:58.204357 5419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:33981 every 8 connection(s)
I20260619 18:58:58.206115 5353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:58:58.211024 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 5353
I20260619 18:58:58.211081 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:58:58.211268 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:58:58.211761 15724 external_mini_cluster.cc:1428] Setting key 0507fc61724373f307bc8e64cc23839c
I20260619 18:58:58.228864 5420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95: Bootstrap starting.
Jun 19 18:58:58 dist-test-slave-dm3f krb5kdc[5340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:58:58.233323 5420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95: Neither blocks nor log segments found. Creating new log.
I20260619 18:58:58.235080 5420 log.cc:824] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95: Log is configured to *not* fsync() on all Append() calls
I20260619 18:58:58.240674 5420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95: No bootstrap required, opened a new log
I20260619 18:58:58.243953 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:58.214040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59209 (local address 127.15.91.62:33981)
0619 18:58:58.215134 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:58.215159 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:58.215232 (+ 73us) server_negotiation.cc:407] Connection header received
0619 18:58:58.217144 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:58.217209 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:58.217963 (+ 754us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:58.218540 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:58.220306 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:58.222331 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:58.223992 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:58.225294 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:58.225572 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:58.231776 (+ 6204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:58.231790 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:58.231812 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:58.231880 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:58.236528 (+ 4648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:58.238217 (+ 1689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:58.238230 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:58.238242 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:58.238591 (+ 349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:58.239234 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:58.239240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:58.239243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:58.240512 (+ 1269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:58.240940 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:58.241710 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 18:58:58.242291 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":430,"threads_started":1}
I20260619 18:58:58.249289 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:58:58.251495 5420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3f3d74f2e54bd4a70595ffef1aad95" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33981 } }
I20260619 18:58:58.252038 5420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:58:58.252171 5420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c3f3d74f2e54bd4a70595ffef1aad95, State: Initialized, Role: FOLLOWER
I20260619 18:58:58.253060 5420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3f3d74f2e54bd4a70595ffef1aad95" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33981 } }
I20260619 18:58:58.253434 5420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:58:58.253623 5420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:58:58.253827 5420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:58:58.257071 5420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3f3d74f2e54bd4a70595ffef1aad95" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33981 } }
I20260619 18:58:58.257862 5420 leader_election.cc:302] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c3f3d74f2e54bd4a70595ffef1aad95; no voters:
I20260619 18:58:58.258656 5420 leader_election.cc:288] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:58:58.258849 5430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:58:58.260288 5430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [term 1 LEADER]: Becoming Leader. State: Replica: 9c3f3d74f2e54bd4a70595ffef1aad95, State: Running, Role: LEADER
I20260619 18:58:58.261222 5430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3f3d74f2e54bd4a70595ffef1aad95" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33981 } }
I20260619 18:58:58.261656 5420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:58:58.269358 5432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c3f3d74f2e54bd4a70595ffef1aad95. Latest consensus state: current_term: 1 leader_uuid: "9c3f3d74f2e54bd4a70595ffef1aad95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3f3d74f2e54bd4a70595ffef1aad95" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33981 } } }
I20260619 18:58:58.269982 5432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:58.270737 5431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c3f3d74f2e54bd4a70595ffef1aad95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3f3d74f2e54bd4a70595ffef1aad95" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33981 } } }
I20260619 18:58:58.271181 5431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95 [sys.catalog]: This master's current role is: LEADER
I20260619 18:58:58.275199 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:58:58.254502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54317 (local address 127.15.91.62:33981)
0619 18:58:58.254761 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 18:58:58.254769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:58:58.255207 (+ 438us) server_negotiation.cc:407] Connection header received
0619 18:58:58.255429 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:58:58.255435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:58:58.255528 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:58:58.255665 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:58:58.257412 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:58.258255 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:58.260402 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:58:58.261168 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:58:58.261316 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:58:58.264964 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:58:58.264971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:58:58.264974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:58:58.265019 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:58:58.269264 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:58.270423 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:58.270432 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:58.270439 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:58.270694 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:58:58.271412 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:58:58.271421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:58:58.271427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:58:58.272048 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:58:58.272461 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0619 18:58:58.274150 (+ 1689us) server_negotiation.cc:299] Negotiation successful
0619 18:58:58.274554 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:58:58.283039 5438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:58:58.290093 5438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:58:58.301208 5438 catalog_manager.cc:1347] Generated new cluster ID: 38d90da4695644ec954f3bf9a92b9c0f
I20260619 18:58:58.301362 5438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:58:58.317369 5438 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:58:58.319571 5438 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:58:58.326794 5438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c3f3d74f2e54bd4a70595ffef1aad95: Generated new TSK 0
I20260619 18:58:58.327893 5438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:58:58.334240 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5353
I20260619 18:58:58.335651 5415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:58:58.430373 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 5353
2026-06-19T18:58:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2143 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:58:58 dist-test-slave-dm3f krb5kdc[5453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:58:58 dist-test-slave-dm3f krb5kdc[5453](info): set up 2 sockets
Jun 19 18:58:58 dist-test-slave-dm3f krb5kdc[5453](info): commencing operation
krb5kdc: starting...
W20260619 18:58:59.555416 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:58:59 dist-test-slave-dm3f krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:58:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:58:59.838472 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:33147
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:33147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:59:00.208041 5466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:59:00.208463 5466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:59:00.208643 5466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:59:00.216441 5466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:59:00.216651 5466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:59:00.216753 5466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:59:00.216877 5466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:59:00.216969 5466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:59:00.227725 5466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:33147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:33147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:59:00.229288 5466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:59:00.230894 5466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:59:00.245013 5474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:00.245026 5472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:00.245086 5471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:59:00.246258 5466 server_base.cc:1034] running on GCE node
I20260619 18:59:00.247365 5466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:59:00.249073 5466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:59:00.250339 5466 hybrid_clock.cc:648] HybridClock initialized: now 1781895540250270 us; error 79 us; skew 500 ppm
I20260619 18:59:00.250828 5466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:59:00 dist-test-slave-dm3f krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895540, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:59:00.254820 5466 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:59:00.256880 5466 webserver.cc:460] Webserver started at http://127.15.91.62:43725/ using document root <none> and password file <none>
I20260619 18:59:00.257802 5466 fs_manager.cc:359] Metadata directory not provided
I20260619 18:59:00.257952 5466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:59:00.258350 5466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:59:00.261998 5466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "b569af6ffce24b8385c64fc6cd58290c"
format_stamp: "Formatted at 2026-06-19 18:59:00 on dist-test-slave-dm3f"
server_key: "fa810c864518ecdb778fa4e198446fa6"
server_key_iv: "af80b919fd83c235f5b2a6cad9f0adc9"
server_key_version: "encryptionkey@0"
I20260619 18:59:00.263082 5466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "b569af6ffce24b8385c64fc6cd58290c"
format_stamp: "Formatted at 2026-06-19 18:59:00 on dist-test-slave-dm3f"
server_key: "fa810c864518ecdb778fa4e198446fa6"
server_key_iv: "af80b919fd83c235f5b2a6cad9f0adc9"
server_key_version: "encryptionkey@0"
I20260619 18:59:00.269783 5466 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:59:00.274477 5481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:59:00.276417 5466 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:59:00.276672 5466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "b569af6ffce24b8385c64fc6cd58290c"
format_stamp: "Formatted at 2026-06-19 18:59:00 on dist-test-slave-dm3f"
server_key: "fa810c864518ecdb778fa4e198446fa6"
server_key_iv: "af80b919fd83c235f5b2a6cad9f0adc9"
server_key_version: "encryptionkey@0"
I20260619 18:59:00.276912 5466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:59:00.317586 5466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:59:00.324246 5466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:59:00.324635 5466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:59:00.347507 5466 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:33147
I20260619 18:59:00.347599 5532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:33147 every 8 connection(s)
I20260619 18:59:00.349161 5466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:59:00.353463 5533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:59:00.357954 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 5466
I20260619 18:59:00.358212 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:59:00.358671 15724 external_mini_cluster.cc:1428] Setting key d0ab26ac6f32c6f15da58ecbb26e458c
I20260619 18:59:00.369395 5533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c: Bootstrap starting.
I20260619 18:59:00.373835 5533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c: Neither blocks nor log segments found. Creating new log.
I20260619 18:59:00.375448 5533 log.cc:824] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:59:00.376411 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:00.360931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40129 (local address 127.15.91.62:33147)
0619 18:59:00.361830 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:00.361848 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:00.361908 (+ 60us) server_negotiation.cc:407] Connection header received
0619 18:59:00.364119 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:00.364170 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:00.364945 (+ 775us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:00.365532 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:00.366209 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:00.368421 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:00.370122 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:00.371303 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:00.371580 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:00.372905 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:00.372917 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:00.372940 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:59:00.373763 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:00.373894 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:59:00.373936 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:00.374073 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:00.374465 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 18:59:00.374855 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":389,"threads_started":1}
I20260619 18:59:00.379359 5533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c: No bootstrap required, opened a new log
I20260619 18:59:00.381027 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:59:00.389359 5533 raft_consensus.cc:348] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569af6ffce24b8385c64fc6cd58290c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33147 } }
I20260619 18:59:00.389987 5533 raft_consensus.cc:374] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:59:00.390105 5533 raft_consensus.cc:729] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b569af6ffce24b8385c64fc6cd58290c, State: Initialized, Role: FOLLOWER
I20260619 18:59:00.390937 5533 consensus_queue.cc:260] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569af6ffce24b8385c64fc6cd58290c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33147 } }
I20260619 18:59:00.391216 5533 raft_consensus.cc:388] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:59:00.391430 5533 raft_consensus.cc:482] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:59:00.391661 5533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:59:00.393651 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:00.386183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34913 (local address 127.15.91.62:33147)
0619 18:59:00.386460 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:00.386471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:00.386912 (+ 441us) server_negotiation.cc:407] Connection header received
0619 18:59:00.387114 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:00.387119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:00.387201 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:00.387308 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:00.387981 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:00.388749 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:00.390809 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:00.391392 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:00.391543 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:00.392821 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:00.392826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:00.392830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:59:00.392858 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:00.392906 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:59:00.392928 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:00.393012 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:00.393169 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 18:59:00.393268 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:59:00.395263 5533 raft_consensus.cc:504] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569af6ffce24b8385c64fc6cd58290c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33147 } }
I20260619 18:59:00.395908 5533 leader_election.cc:302] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b569af6ffce24b8385c64fc6cd58290c; no voters:
I20260619 18:59:00.396569 5533 leader_election.cc:288] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:59:00.396744 5543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:59:00.397630 5543 raft_consensus.cc:686] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [term 1 LEADER]: Becoming Leader. State: Replica: b569af6ffce24b8385c64fc6cd58290c, State: Running, Role: LEADER
I20260619 18:59:00.398216 5543 consensus_queue.cc:237] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569af6ffce24b8385c64fc6cd58290c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33147 } }
I20260619 18:59:00.399084 5533 sys_catalog.cc:564] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:59:00.402319 5545 sys_catalog.cc:455] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b569af6ffce24b8385c64fc6cd58290c. Latest consensus state: current_term: 1 leader_uuid: "b569af6ffce24b8385c64fc6cd58290c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569af6ffce24b8385c64fc6cd58290c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33147 } } }
I20260619 18:59:00.402876 5545 sys_catalog.cc:458] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [sys.catalog]: This master's current role is: LEADER
I20260619 18:59:00.404130 5552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:59:00.403945 5544 sys_catalog.cc:455] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b569af6ffce24b8385c64fc6cd58290c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569af6ffce24b8385c64fc6cd58290c" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 33147 } } }
I20260619 18:59:00.404397 5544 sys_catalog.cc:458] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c [sys.catalog]: This master's current role is: LEADER
I20260619 18:59:00.411015 5552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:59:00.422209 5552 catalog_manager.cc:1347] Generated new cluster ID: a9236235c389482eb4df5e43a77bc1a9
I20260619 18:59:00.422374 5552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:59:00.438887 5552 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:59:00.440399 5552 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:59:00.450037 5552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b569af6ffce24b8385c64fc6cd58290c: Generated new TSK 0
I20260619 18:59:00.451140 5552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:59:00.457329 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5466
W20260619 18:59:00.458684 5528 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40129
W20260619 18:59:00.459637 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5466: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:59:00.459810 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 5466
2026-06-19T18:59:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:59:00 dist-test-slave-dm3f krb5kdc[5564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:59:00 dist-test-slave-dm3f krb5kdc[5564](info): set up 2 sockets
Jun 19 18:59:00 dist-test-slave-dm3f krb5kdc[5564](info): commencing operation
krb5kdc: starting...
W20260619 18:59:01.603479 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:59:01 dist-test-slave-dm3f krb5kdc[5564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:59:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:59:01.898849 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:34791
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:38931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:34791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:59:02.267100 5577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:59:02.267438 5577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:59:02.267612 5577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:59:02.275556 5577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:59:02.275691 5577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:59:02.275812 5577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:59:02.275933 5577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:59:02.275998 5577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:59:02.286556 5577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:38931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:34791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:34791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:59:02.288084 5577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:59:02.289541 5577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:59:02.302995 5582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:02.302940 5585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:02.303018 5583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:59:02.304013 5577 server_base.cc:1034] running on GCE node
I20260619 18:59:02.305022 5577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:59:02.306650 5577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:59:02.308014 5577 hybrid_clock.cc:648] HybridClock initialized: now 1781895542307939 us; error 73 us; skew 500 ppm
I20260619 18:59:02.308516 5577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:59:02 dist-test-slave-dm3f krb5kdc[5564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895542, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:59:02.312315 5577 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:59:02.314224 5577 webserver.cc:460] Webserver started at http://127.15.91.62:42739/ using document root <none> and password file <none>
I20260619 18:59:02.315120 5577 fs_manager.cc:359] Metadata directory not provided
I20260619 18:59:02.315267 5577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:59:02.315567 5577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:59:02.319190 5577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "67c3c7f16f5346ca8ed62a25b58613a5"
format_stamp: "Formatted at 2026-06-19 18:59:02 on dist-test-slave-dm3f"
server_key: "799c60c098061c1e06985d5b57b49f30"
server_key_iv: "26d3d22932fd9ebb12669234847c3863"
server_key_version: "encryptionkey@0"
I20260619 18:59:02.320086 5577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "67c3c7f16f5346ca8ed62a25b58613a5"
format_stamp: "Formatted at 2026-06-19 18:59:02 on dist-test-slave-dm3f"
server_key: "799c60c098061c1e06985d5b57b49f30"
server_key_iv: "26d3d22932fd9ebb12669234847c3863"
server_key_version: "encryptionkey@0"
I20260619 18:59:02.326396 5577 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:59:02.330837 5592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:59:02.332532 5577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:59:02.332791 5577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "67c3c7f16f5346ca8ed62a25b58613a5"
format_stamp: "Formatted at 2026-06-19 18:59:02 on dist-test-slave-dm3f"
server_key: "799c60c098061c1e06985d5b57b49f30"
server_key_iv: "26d3d22932fd9ebb12669234847c3863"
server_key_version: "encryptionkey@0"
I20260619 18:59:02.333046 5577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:59:02.339798 5577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:59:02.344969 5577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:59:02.345348 5577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:59:02.366539 5577 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:34791
I20260619 18:59:02.366582 5643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:34791 every 8 connection(s)
I20260619 18:59:02.368366 5577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:59:02.372788 5644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:59:02.377595 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 5577
I20260619 18:59:02.377815 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:59:02.378378 15724 external_mini_cluster.cc:1428] Setting key 53b64aeab22c36342cb277717d9eb51a
I20260619 18:59:02.387773 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:02.380597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52669 (local address 127.15.91.62:34791)
0619 18:59:02.381593 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:02.381608 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:02.381655 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:59:02.383728 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:02.383762 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:02.384229 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:02.384647 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:02.385869 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:02.385877 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:02.385894 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:59:02.385911 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:02.385975 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:59:02.385997 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:02.386201 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:02.386352 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 18:59:02.386612 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":451,"threads_started":1}
I20260619 18:59:02.388525 5644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5: Bootstrap starting.
I20260619 18:59:02.392163 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:59:02.392179 5644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5: Neither blocks nor log segments found. Creating new log.
I20260619 18:59:02.393611 5644 log.cc:824] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:59:02.397783 5644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5: No bootstrap required, opened a new log
I20260619 18:59:02.400627 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:02.397706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50129 (local address 127.15.91.62:34791)
0619 18:59:02.397959 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:02.397966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:02.398353 (+ 387us) server_negotiation.cc:407] Connection header received
0619 18:59:02.398599 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:02.398604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:02.398661 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:02.398860 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:02.399868 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:02.399873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:02.399876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:59:02.399893 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:02.399934 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:59:02.399950 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:02.400050 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:02.400206 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 18:59:02.400265 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:59:02.407611 5644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c3c7f16f5346ca8ed62a25b58613a5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34791 } }
I20260619 18:59:02.408092 5644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:59:02.408229 5644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c3c7f16f5346ca8ed62a25b58613a5, State: Initialized, Role: FOLLOWER
I20260619 18:59:02.408890 5644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c3c7f16f5346ca8ed62a25b58613a5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34791 } }
I20260619 18:59:02.409225 5644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:59:02.409399 5644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:59:02.409581 5644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:59:02.412567 5644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c3c7f16f5346ca8ed62a25b58613a5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34791 } }
I20260619 18:59:02.413226 5644 leader_election.cc:302] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67c3c7f16f5346ca8ed62a25b58613a5; no voters:
I20260619 18:59:02.413928 5644 leader_election.cc:288] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:59:02.414117 5654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:59:02.414961 5654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [term 1 LEADER]: Becoming Leader. State: Replica: 67c3c7f16f5346ca8ed62a25b58613a5, State: Running, Role: LEADER
I20260619 18:59:02.415690 5654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c3c7f16f5346ca8ed62a25b58613a5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34791 } }
I20260619 18:59:02.416399 5644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:59:02.419047 5655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67c3c7f16f5346ca8ed62a25b58613a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c3c7f16f5346ca8ed62a25b58613a5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34791 } } }
I20260619 18:59:02.419561 5655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:59:02.420759 5662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:59:02.421144 5656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67c3c7f16f5346ca8ed62a25b58613a5. Latest consensus state: current_term: 1 leader_uuid: "67c3c7f16f5346ca8ed62a25b58613a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c3c7f16f5346ca8ed62a25b58613a5" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 34791 } } }
I20260619 18:59:02.421572 5656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:59:02.428167 5662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:59:02.439494 5662 catalog_manager.cc:1347] Generated new cluster ID: 984edeeae4b243c6b8f878135b812bdc
I20260619 18:59:02.439635 5662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:59:02.476827 5662 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:59:02.477409 5662 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:59:02.490429 5662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67c3c7f16f5346ca8ed62a25b58613a5: Generated new TSK 0
I20260619 18:59:02.491519 5662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:59:02.499456 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5577
W20260619 18:59:02.501250 5639 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52669
W20260619 18:59:02.502162 15724 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5577: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:59:02.502336 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 5577
2026-06-19T18:59:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2045 ms)
[----------] 18 tests from AuthTokenIssuingTest (37388 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:59:02 dist-test-slave-dm3f krb5kdc[5675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:59:02 dist-test-slave-dm3f krb5kdc[5675](info): set up 2 sockets
Jun 19 18:59:02 dist-test-slave-dm3f krb5kdc[5675](info): commencing operation
krb5kdc: starting...
W20260619 18:59:03.623565 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:59:03 dist-test-slave-dm3f krb5kdc[5675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:59:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:59:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:59:03.906973 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:39627
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:39627,127.15.91.61:44539,127.15.91.60:44497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:59:04.271773 5688 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:59:04.272195 5688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:59:04.272354 5688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:59:04.272507 5688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:59:04.280638 5688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:59:04.280766 5688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:59:04.280870 5688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:59:04.280953 5688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:59:04.281069 5688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:59:04.291738 5688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:39627,127.15.91.61:44539,127.15.91.60:44497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:39627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:59:04.293284 5688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:59:04.294919 5688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:59:04.308938 5693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:04.308984 5694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:04.309052 5696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:59:04.310211 5688 server_base.cc:1034] running on GCE node
I20260619 18:59:04.311234 5688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:59:04.313012 5688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:59:04.314491 5688 hybrid_clock.cc:648] HybridClock initialized: now 1781895544314283 us; error 115 us; skew 500 ppm
I20260619 18:59:04.315341 5688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:59:04 dist-test-slave-dm3f krb5kdc[5675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:59:04.322268 5688 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:59:04.324438 5688 webserver.cc:460] Webserver started at http://127.15.91.62:35043/ using document root <none> and password file <none>
I20260619 18:59:04.325323 5688 fs_manager.cc:359] Metadata directory not provided
I20260619 18:59:04.325498 5688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:59:04.325876 5688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:59:04.329424 5688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "4163a3cbf274424d90bfa4e55864ea6b"
format_stamp: "Formatted at 2026-06-19 18:59:04 on dist-test-slave-dm3f"
server_key: "e55c2e8fa59d85aed8193a2aebe2eb5a"
server_key_iv: "c0943142bb7db6699d5e16ac3aa8a088"
server_key_version: "encryptionkey@0"
I20260619 18:59:04.330435 5688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "4163a3cbf274424d90bfa4e55864ea6b"
format_stamp: "Formatted at 2026-06-19 18:59:04 on dist-test-slave-dm3f"
server_key: "e55c2e8fa59d85aed8193a2aebe2eb5a"
server_key_iv: "c0943142bb7db6699d5e16ac3aa8a088"
server_key_version: "encryptionkey@0"
I20260619 18:59:04.336561 5688 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260619 18:59:04.340977 5703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:59:04.342602 5688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:59:04.342902 5688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "4163a3cbf274424d90bfa4e55864ea6b"
format_stamp: "Formatted at 2026-06-19 18:59:04 on dist-test-slave-dm3f"
server_key: "e55c2e8fa59d85aed8193a2aebe2eb5a"
server_key_iv: "c0943142bb7db6699d5e16ac3aa8a088"
server_key_version: "encryptionkey@0"
I20260619 18:59:04.343186 5688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:59:04.358695 5688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:59:04.363270 5688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:59:04.363610 5688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:59:04.385107 5688 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:39627
I20260619 18:59:04.385118 5754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:39627 every 8 connection(s)
I20260619 18:59:04.386703 5688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:59:04.391368 5755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:59:04.396178 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 5688
I20260619 18:59:04.396422 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:59:04.397260 15724 external_mini_cluster.cc:1428] Setting key cf7604a58fb7af84f2331000c1c8c170
I20260619 18:59:04.401218 5755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:59:04 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.61@KRBTEST.COM" created.
I20260619 18:59:04.432615 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.406054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39627 (local address 127.0.0.1:55992)
0619 18:59:04.407666 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.407719 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:04.410540 (+ 2821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:04.411830 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:04.411841 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:04.412831 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:04.413429 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:04.413465 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:04.415509 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:04.415518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:04.417727 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:04.417758 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:04.419085 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:04.419097 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:04.419345 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:04.420156 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:04.420195 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:04.423589 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:04.427852 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:04.427870 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:04.427889 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:04.428398 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:04.428968 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:04.428973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:04.428983 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:04.429142 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:04.430258 (+ 1116us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:04.430266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:04.430492 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 18:59:04.430863 (+ 371us) client_negotiation.cc:240] Negotiation successful
0619 18:59:04.431352 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"spinlock_wait_cycles":29568,"thread_start_us":1021,"threads_started":1}
I20260619 18:59:04.432651 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.406054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.15.91.62:39627)
0619 18:59:04.407200 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:04.407222 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:04.407925 (+ 703us) server_negotiation.cc:407] Connection header received
0619 18:59:04.411166 (+ 3241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:04.411176 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:04.411500 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:04.411714 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:04.413777 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:04.415279 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:04.417998 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:04.418660 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:04.418948 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:04.423924 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:04.423934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:04.423950 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:04.423989 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:04.427569 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:04.428562 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:04.428578 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:04.428580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:04.428817 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:04.429301 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:04.429309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:04.429323 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:04.430039 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:04.430269 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:04.431023 (+ 754us) server_negotiation.cc:299] Negotiation successful
0619 18:59:04.431595 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":184,"server-negotiator.queue_time_us":816,"thread_start_us":505,"threads_started":1}
I20260619 18:59:04.436432 5755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:59:04.437790 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.437298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53098)
0619 18:59:04.437508 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.437636 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 18:59:04.438161 5706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
I20260619 18:59:04.440104 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.439625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53100)
0619 18:59:04.439843 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.439937 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 18:59:04.440361 5755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:44539: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab.
Entry for principal kudu/127.15.91.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab.
I20260619 18:59:04.464112 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.61:44539
--webserver_interface=127.15.91.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab
--principal=kudu/127.15.91.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:39627,127.15.91.61:44539,127.15.91.60:44497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:59:04.489765 5755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } attempt: 1
I20260619 18:59:04.491011 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.490509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53102)
0619 18:59:04.490738 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.490854 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 18:59:04.492249 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.491805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53104)
0619 18:59:04.492007 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.492088 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 18:59:04.492463 5755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:44539: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
I20260619 18:59:04.560742 5755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } attempt: 2
I20260619 18:59:04.562088 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.561465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53116)
0619 18:59:04.561767 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.561888 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 18:59:04.563407 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.562904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53120)
0619 18:59:04.563077 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.563198 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 18:59:04.563645 5755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:44539: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
I20260619 18:59:04.654959 5755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } attempt: 3
I20260619 18:59:04.656376 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.655725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53130)
0619 18:59:04.656008 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.656145 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 18:59:04.657727 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.657166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53132)
0619 18:59:04.657425 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.657532 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260619 18:59:04.657959 5755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:44539: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
I20260619 18:59:04.801268 5755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } attempt: 4
I20260619 18:59:04.802800 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.802080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53138)
0619 18:59:04.802444 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.802564 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260619 18:59:04.804093 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.803601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53154)
0619 18:59:04.803788 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.803911 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:59:04.804390 5755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:44539: Network error: Client connection negotiation failed: client connection to 127.15.91.61:44539: connect: Connection refused (error 111)
W20260619 18:59:04.832969 5762 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:59:04.833348 5762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:59:04.833422 5762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:59:04.833508 5762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:59:04.841419 5762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:59:04.841533 5762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:59:04.841593 5762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:59:04.841646 5762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:59:04.841711 5762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:59:04.852571 5762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:39627,127.15.91.61:44539,127.15.91.60:44497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab
--principal=kudu/127.15.91.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.61:44539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.91.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:59:04.853960 5762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:59:04.855408 5762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:59:04.868831 5768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:04.868911 5767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:04.868906 5770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:59:04.870138 5762 server_base.cc:1034] running on GCE node
I20260619 18:59:04.871197 5762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:59:04.872916 5762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:59:04.874238 5762 hybrid_clock.cc:648] HybridClock initialized: now 1781895544874151 us; error 64 us; skew 500 ppm
I20260619 18:59:04.874665 5762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:59:04 dist-test-slave-dm3f krb5kdc[5675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:59:04.878613 5762 init.cc:377] Logged in from keytab as kudu/127.15.91.61@KRBTEST.COM (short username kudu)
I20260619 18:59:04.880559 5762 webserver.cc:460] Webserver started at http://127.15.91.61:40661/ using document root <none> and password file <none>
I20260619 18:59:04.881412 5762 fs_manager.cc:359] Metadata directory not provided
I20260619 18:59:04.881570 5762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:59:04.881946 5762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:59:04.885383 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/data/instance:
uuid: "eb911dca77ae4249a22f089b406953b5"
format_stamp: "Formatted at 2026-06-19 18:59:04 on dist-test-slave-dm3f"
server_key: "173f30840c6637116d92f8fcb392e2df"
server_key_iv: "7df909a0ad38c6df5684acb002cdeeba"
server_key_version: "encryptionkey@0"
I20260619 18:59:04.886368 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/wal/instance:
uuid: "eb911dca77ae4249a22f089b406953b5"
format_stamp: "Formatted at 2026-06-19 18:59:04 on dist-test-slave-dm3f"
server_key: "173f30840c6637116d92f8fcb392e2df"
server_key_iv: "7df909a0ad38c6df5684acb002cdeeba"
server_key_version: "encryptionkey@0"
I20260619 18:59:04.892760 5762 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260619 18:59:04.897159 5777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:59:04.898891 5762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:59:04.899159 5762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/wal
uuid: "eb911dca77ae4249a22f089b406953b5"
format_stamp: "Formatted at 2026-06-19 18:59:04 on dist-test-slave-dm3f"
server_key: "173f30840c6637116d92f8fcb392e2df"
server_key_iv: "7df909a0ad38c6df5684acb002cdeeba"
server_key_version: "encryptionkey@0"
I20260619 18:59:04.899454 5762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:59:04.921326 5762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:59:04.926604 5762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:59:04.926973 5762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:59:04.948789 5762 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.61:44539
I20260619 18:59:04.948809 5828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.61:44539 every 8 connection(s)
I20260619 18:59:04.950383 5762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
I20260619 18:59:04.952775 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 5762
I20260619 18:59:04.952997 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-1/wal/instance
I20260619 18:59:04.953588 15724 external_mini_cluster.cc:1428] Setting key 3d151aae264c1d3b47b8d2d699b8c8f5
I20260619 18:59:04.955662 5829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:59:04.966352 5829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.91.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.60@KRBTEST.COM" created.
Jun 19 18:59:04 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:59:04.993362 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.969541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.15.91.62:39627)
0619 18:59:04.970103 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:04.970108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:04.971454 (+ 1346us) server_negotiation.cc:407] Connection header received
0619 18:59:04.973688 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:04.973694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:04.973804 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:04.973937 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:04.976111 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:04.976929 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:04.978789 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:04.979239 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:04.979415 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:04.986281 (+ 6866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:04.986286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:04.986290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:04.986344 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:04.989417 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:04.990426 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:04.990432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:04.990434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:04.990575 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:04.991157 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:04.991161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:04.991163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:04.991505 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:04.991679 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:04.992831 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0619 18:59:04.993053 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":319,"threads_started":1}
I20260619 18:59:04.994580 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.970318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39627 (local address 127.0.0.1:56000)
0619 18:59:04.971250 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.971299 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:04.973299 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:04.974254 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:04.974264 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:04.975015 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:04.975779 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:04.975813 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:04.977153 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:04.977158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:04.978536 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:04.978556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:04.979552 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:04.979561 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:04.979842 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:04.980740 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:04.980793 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:04.985920 (+ 5127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:04.989648 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:04.989660 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:04.989690 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:04.990236 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:04.990746 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:04.990754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:04.990757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:04.991001 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:04.991662 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:04.991672 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:04.992296 (+ 624us) client_negotiation.cc:769] Sending connection context
0619 18:59:04.992673 (+ 377us) client_negotiation.cc:240] Negotiation successful
0619 18:59:04.993334 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":423,"threads_started":1}
I20260619 18:59:04.996842 5829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:59:05 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:59:05.015938 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.997639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53160)
0619 18:59:04.997856 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:04.997877 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:04.997999 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:04.999348 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:04.999352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:04.999769 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:04.999918 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:04.999927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.002170 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.002174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.003168 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.003207 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.004809 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.004814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.004953 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.005651 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:05.005672 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:05.008337 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:05.012907 (+ 4570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.012912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.012915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.013464 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.013992 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.013995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.013997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.014145 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.014908 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:05.014950 (+ 42us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:05.015121 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 18:59:05.015247 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:59:05.015592 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 18:59:05.016284 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:04.997943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.15.91.61:44539)
0619 18:59:04.998655 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:04.998667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:04.998705 (+ 38us) server_negotiation.cc:407] Connection header received
0619 18:59:04.998816 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:04.998829 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:04.999122 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:04.999313 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.000132 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.001951 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.003394 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.004569 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.004773 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.008674 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.008701 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.008739 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.008803 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.012670 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.013649 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.013662 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.013668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.013871 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.014252 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.014256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.014258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.014699 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.014966 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.015398 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.015751 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":488,"thread_start_us":326,"threads_started":1}
I20260619 18:59:05.019061 5829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab.
Entry for principal kudu/127.15.91.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab.
I20260619 18:59:05.020645 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.019969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44206)
0619 18:59:05.020214 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.020418 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 18:59:05.020665 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.60:44497
--webserver_interface=127.15.91.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab
--principal=kudu/127.15.91.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:39627,127.15.91.61:44539,127.15.91.60:44497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:59:05.021108 5778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
I20260619 18:59:05.023154 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.022571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44220)
0619 18:59:05.022834 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.022959 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 18:59:05.023562 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:44497: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
I20260619 18:59:05.073009 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } attempt: 1
I20260619 18:59:05.074288 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.073788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44230)
0619 18:59:05.074012 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.074114 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 18:59:05.075815 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.075289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44236)
0619 18:59:05.075506 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.075660 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 18:59:05.076133 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:44497: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
I20260619 18:59:05.103663 5755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } attempt: 5
Jun 19 18:59:05 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:59:05.120855 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.104506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53174)
0619 18:59:05.104750 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.104774 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:05.104939 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:05.105362 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:05.105368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:05.105921 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:05.106224 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.106241 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.107484 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.107493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.109086 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.109109 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.109976 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.109982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.110231 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.111033 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:05.111059 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:05.113729 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:05.117979 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.117984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.117987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.118478 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.119048 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.119052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.119054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.119206 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.120008 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:05.120011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:05.120172 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 18:59:05.120305 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 18:59:05.120585 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 18:59:05.121048 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.104569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.15.91.61:44539)
0619 18:59:05.104811 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.104816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.104852 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:59:05.105081 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.105086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.105170 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.105265 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.106452 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.107287 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.109313 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.109775 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.109893 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.114043 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.114051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.114056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.114106 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.117717 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.118656 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.118663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.118666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.118895 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.119345 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.119353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.119358 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.119849 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.120015 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.120456 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.120717 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 18:59:05.122255 5755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:59:05.123641 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.123204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44242)
0619 18:59:05.123388 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.123477 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 18:59:05.124845 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.124368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44248)
0619 18:59:05.124526 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.124656 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 18:59:05.125206 5755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:44497: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
I20260619 18:59:05.144413 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } attempt: 2
I20260619 18:59:05.145768 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.145137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44260)
0619 18:59:05.145425 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.145551 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260619 18:59:05.147148 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.146604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44272)
0619 18:59:05.146814 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.146939 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 18:59:05.147462 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:44497: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
I20260619 18:59:05.176518 5755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } attempt: 1
I20260619 18:59:05.177973 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.177412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44278)
0619 18:59:05.177659 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.177775 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 18:59:05.179312 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.178858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44280)
0619 18:59:05.179026 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.179148 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 18:59:05.179714 5755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:44497: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
I20260619 18:59:05.238752 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } attempt: 3
I20260619 18:59:05.240150 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.239527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44284)
0619 18:59:05.239800 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.239930 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 18:59:05.241545 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.240987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44294)
0619 18:59:05.241251 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.241372 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260619 18:59:05.241860 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:44497: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
I20260619 18:59:05.248021 5755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } attempt: 2
I20260619 18:59:05.249287 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.248757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44302)
0619 18:59:05.249027 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.249116 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 18:59:05.250336 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.249972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44316)
0619 18:59:05.250127 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.250209 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 18:59:05.250553 5755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:44497: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
I20260619 18:59:05.356760 5755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } attempt: 3
I20260619 18:59:05.358060 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.357573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44332)
0619 18:59:05.357792 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.357885 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 18:59:05.359308 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.358860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44334)
0619 18:59:05.359091 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.359174 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260619 18:59:05.359512 5755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:44497: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
I20260619 18:59:05.385119 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } attempt: 4
I20260619 18:59:05.387032 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.386072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44344)
0619 18:59:05.386645 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.386797 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315}
I20260619 18:59:05.390110 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.389124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44350)
0619 18:59:05.389807 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.389919 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540}
W20260619 18:59:05.390381 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:44497: Network error: Client connection negotiation failed: client connection to 127.15.91.60:44497: connect: Connection refused (error 111)
W20260619 18:59:05.402899 5837 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:59:05.403280 5837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:59:05.403384 5837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:59:05.403523 5837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:59:05.411499 5837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:59:05.411613 5837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:59:05.411715 5837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:59:05.411803 5837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:59:05.411917 5837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:59:05.422680 5837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:39627,127.15.91.61:44539,127.15.91.60:44497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab
--principal=kudu/127.15.91.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.60:44497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.91.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:59:05.424229 5837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:59:05.425714 5837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:59:05.439409 5843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:05.439316 5842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:05.439342 5845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:59:05.440970 5837 server_base.cc:1034] running on GCE node
I20260619 18:59:05.441990 5837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:59:05.443946 5837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:59:05.445245 5837 hybrid_clock.cc:648] HybridClock initialized: now 1781895545445136 us; error 75 us; skew 500 ppm
I20260619 18:59:05.445694 5837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:59:05 dist-test-slave-dm3f krb5kdc[5675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895545, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:59:05.449528 5837 init.cc:377] Logged in from keytab as kudu/127.15.91.60@KRBTEST.COM (short username kudu)
I20260619 18:59:05.451368 5837 webserver.cc:460] Webserver started at http://127.15.91.60:35267/ using document root <none> and password file <none>
I20260619 18:59:05.452157 5837 fs_manager.cc:359] Metadata directory not provided
I20260619 18:59:05.452260 5837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:59:05.452566 5837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:59:05.456012 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/data/instance:
uuid: "3ee9dd3d05784bca89fbdf443f6f52d1"
format_stamp: "Formatted at 2026-06-19 18:59:05 on dist-test-slave-dm3f"
server_key: "512a83057909135bfdb8e33fadebf712"
server_key_iv: "78aab90d0cb212540edd4db6fc03fa3a"
server_key_version: "encryptionkey@0"
I20260619 18:59:05.456933 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/wal/instance:
uuid: "3ee9dd3d05784bca89fbdf443f6f52d1"
format_stamp: "Formatted at 2026-06-19 18:59:05 on dist-test-slave-dm3f"
server_key: "512a83057909135bfdb8e33fadebf712"
server_key_iv: "78aab90d0cb212540edd4db6fc03fa3a"
server_key_version: "encryptionkey@0"
I20260619 18:59:05.463639 5837 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 18:59:05.467892 5852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:59:05.469635 5837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:59:05.469844 5837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/wal
uuid: "3ee9dd3d05784bca89fbdf443f6f52d1"
format_stamp: "Formatted at 2026-06-19 18:59:05 on dist-test-slave-dm3f"
server_key: "512a83057909135bfdb8e33fadebf712"
server_key_iv: "78aab90d0cb212540edd4db6fc03fa3a"
server_key_version: "encryptionkey@0"
I20260619 18:59:05.470077 5837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:59:05.493407 5837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:59:05.498333 5837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:59:05.498597 5837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:59:05.520879 5837 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.60:44497
I20260619 18:59:05.520900 5903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.60:44497 every 8 connection(s)
I20260619 18:59:05.522536 5837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
I20260619 18:59:05.527232 5904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:59:05.530866 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 5837
I20260619 18:59:05.531086 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781895164554191-15724-0/minicluster-data/master-2/wal/instance
I20260619 18:59:05.531626 15724 external_mini_cluster.cc:1428] Setting key 7b00a92f53233971d792c91587c1dd38
I20260619 18:59:05.536777 5755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } attempt: 4
I20260619 18:59:05.537027 5904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:59:05 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
Jun 19 18:59:05 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
Jun 19 18:59:05 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895545, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:59:05.559806 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.537560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44372)
0619 18:59:05.537849 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.537873 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:05.537993 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:05.539154 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:05.539159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:05.539702 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:05.539867 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.539878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.543328 (+ 3450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.543334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.544858 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.544880 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.546102 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.546109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.546349 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.547274 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:05.547300 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:05.550765 (+ 3465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:05.556078 (+ 5313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.556084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.556087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.556739 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.557279 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.557287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.557293 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.557487 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.558538 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:05.558544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:05.558779 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 18:59:05.559015 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 18:59:05.559330 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260619 18:59:05.561308 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.534096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.15.91.60:44497)
0619 18:59:05.535133 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.535149 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.535204 (+ 55us) server_negotiation.cc:407] Connection header received
0619 18:59:05.537398 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.537438 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.538166 (+ 728us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.538759 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.540787 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.543108 (+ 2321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.544632 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.545878 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.547006 (+ 1128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.551794 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.551831 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.551838 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.551892 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.555789 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.556841 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.556853 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.556866 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.557122 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.557660 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.557665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.557668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.558333 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.558680 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.559232 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.559756 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":379,"server-negotiator.queue_time_us":778,"thread_start_us":459,"threads_started":1}
I20260619 18:59:05.562040 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.541448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:39627 (local address 127.0.0.1:56002)
0619 18:59:05.542569 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.542619 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:05.542923 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:05.543589 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:05.543598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:05.544580 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:05.545004 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.545020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.546099 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.546104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.547731 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.547756 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.549496 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.549508 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.549787 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.550623 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:05.550651 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:05.554326 (+ 3675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:05.558746 (+ 4420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.558759 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.558766 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.559381 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.559852 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.559857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.559863 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.560052 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.560754 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:05.560764 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:05.561017 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 18:59:05.561273 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 18:59:05.561653 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":672,"threads_started":1}
I20260619 18:59:05.562135 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.541227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.15.91.62:39627)
0619 18:59:05.541903 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.541912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.542769 (+ 857us) server_negotiation.cc:407] Connection header received
0619 18:59:05.543103 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.543109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.543198 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.543344 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.545135 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.545864 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.548022 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.548932 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.549127 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.554672 (+ 5545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.554682 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.554688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.554765 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.558560 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.559536 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.559542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.559546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.559690 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.560209 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.560216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.560223 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.560596 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.560778 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.561452 (+ 674us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.561734 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":343,"threads_started":1}
I20260619 18:59:05.561333 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.537822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.15.91.60:44497)
0619 18:59:05.538638 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.538644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.538805 (+ 161us) server_negotiation.cc:407] Connection header received
0619 18:59:05.538869 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.538877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.538956 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.540008 (+ 1052us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.540289 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.543108 (+ 2819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.545051 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.545878 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.546963 (+ 1085us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.549580 (+ 2617us) spinlock_profiling.cc:243] Waited 1.05 ms on lock 0x60c0000347b8. stack: 00007fce3dcb166e 00007fce3dcb1430 00007fce3cd385ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fce3dca7824 00007fce3dca6d66 00007fce447be664 00007fce447d8e8d 00007fce447d3f76 00007fce447cd8a5 00007fce4473bea5 00007fce4473aeff 00007fce4476e82f 00007fce4476e58d
0619 18:59:05.551018 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.551029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.551050 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.551093 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.555928 (+ 4835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.556901 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.556905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.556908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.557100 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.557648 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.557654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.557658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.558351 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.560374 (+ 2023us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.560496 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.560713 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":351,"server-negotiator.queue_time_us":634,"spinlock_wait_cycles":2444160,"thread_start_us":429,"threads_started":1}
I20260619 18:59:05.563851 5904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:59:05.569805 5755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b: Bootstrap starting.
Jun 19 18:59:05 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895545, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:59:05.574504 5755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b: Neither blocks nor log segments found. Creating new log.
I20260619 18:59:05.576064 5755 log.cc:824] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:59:05 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:59:05.581292 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.564644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:44539 (local address 127.0.0.1:53178)
0619 18:59:05.564848 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.564866 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:05.565005 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:05.565978 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:05.565983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:05.566538 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:05.566791 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.566811 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.568324 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.568331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.569530 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.569548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.570480 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.570487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.570694 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.571493 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:05.571519 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:05.574111 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:05.578344 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.578350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.578354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.578911 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.579428 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.579434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.579440 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.579608 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.580235 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:05.580240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:05.580414 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 18:59:05.580560 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 18:59:05.580909 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 18:59:05.581327 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.565251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53178 (local address 127.15.91.61:44539)
0619 18:59:05.565457 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.565464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.565484 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:59:05.565554 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.565559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.565650 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.565759 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.566999 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.568102 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.569718 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.570287 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.570456 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.574372 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.574378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.574381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.574422 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.578168 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.579061 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.579074 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.579077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.579232 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.579759 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.579764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.579766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.580083 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.580261 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.580689 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.580922 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:59:05.582801 5904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:59:05.586637 5755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b: No bootstrap required, opened a new log
I20260619 18:59:05.587410 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.565580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56014 (local address 127.15.91.62:39627)
0619 18:59:05.565797 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.565803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.565899 (+ 96us) server_negotiation.cc:407] Connection header received
0619 18:59:05.566101 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.566105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.566206 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.566330 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.567817 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.568882 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.570694 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.571504 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.571674 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.577910 (+ 6236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.577921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.577930 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.577998 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.584203 (+ 6205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.585123 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.585129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.585133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.585295 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.585713 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.585719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.585722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.586042 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.586379 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.586769 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.587049 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 18:59:05 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895545, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
I20260619 18:59:05.598341 5755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } }
I20260619 18:59:05.598809 5755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4163a3cbf274424d90bfa4e55864ea6b, State: Initialized, Role: FOLLOWER
I20260619 18:59:05.599354 5755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } }
I20260619 18:59:05.599573 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.583726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44386)
0619 18:59:05.583932 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.583954 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:05.584086 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:05.584629 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:05.584633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:05.585124 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:05.585345 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.585358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.586704 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.586739 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.588079 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.588100 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.588982 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.588987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.589259 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.589972 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:05.590000 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:05.592333 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:05.595822 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.595828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.595834 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.596450 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.597350 (+ 900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.597355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.597360 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.597539 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.598309 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:05.598314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:05.598524 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 18:59:05.598748 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 18:59:05.599164 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":44}
Jun 19 18:59:05 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:59:05.600607 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.583725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.15.91.60:44497)
0619 18:59:05.584285 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.584290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.584331 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:59:05.584383 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.584394 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.584463 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.584563 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.585517 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.586538 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.588265 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.588809 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.588971 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.592539 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.592545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.592551 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.592590 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.595635 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.596608 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.596922 (+ 314us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.596933 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.597167 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.597706 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.597711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.597714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.598149 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.598337 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.599969 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.600237 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":466}
I20260619 18:59:05.601033 5912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } } }
I20260619 18:59:05.601804 5912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:05.602258 5755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:59:05.607086 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1: Bootstrap starting.
I20260619 18:59:05.611882 5904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1: Neither blocks nor log segments found. Creating new log.
I20260619 18:59:05.612389 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.589561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.15.91.61:44539)
0619 18:59:05.589768 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.589775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.589796 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:59:05.589951 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.589958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.590041 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.590155 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.592004 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.593164 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.595246 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.595950 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.596107 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.603184 (+ 7077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.603194 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.603200 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.603251 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.607472 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.608773 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.608780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.608785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.609050 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.609713 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.609718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.609722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.610170 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.610534 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.611566 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.611857 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:59:05.613476 5904 log.cc:824] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1: Log is configured to *not* fsync() on all Append() calls
W20260619 18:59:05.616899 5709 tablet.cc:2369] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:59:05.617726 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1: No bootstrap required, opened a new log
I20260619 18:59:05.625442 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 18:59:05.626106 5923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:59:05.626264 5923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:59:05.627332 5904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } }
I20260619 18:59:05.627872 5904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee9dd3d05784bca89fbdf443f6f52d1, State: Initialized, Role: FOLLOWER
I20260619 18:59:05.628504 5904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } }
I20260619 18:59:05.629457 5730 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4163a3cbf274424d90bfa4e55864ea6b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56014
I20260619 18:59:05.629964 5730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 18:59:05.630189 5730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:59:05.631284 5926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } } }
I20260619 18:59:05.631737 5926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:05.632936 5904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:59:05.634532 5730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } }
I20260619 18:59:05.616267 5688 master_runner.cc:186] Error getting master registration for 127.15.91.61:44539: OK, instance_id { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" instance_seqno: 1781895544942483 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:59:05.637812 5688 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:59:05.638594 5730 leader_election.cc:288] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [CANDIDATE]: Term 1 election: Requested vote from peers eb911dca77ae4249a22f089b406953b5 (127.15.91.61:44539), 3ee9dd3d05784bca89fbdf443f6f52d1 (127.15.91.60:44497)
I20260619 18:59:05.639366 5879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4163a3cbf274424d90bfa4e55864ea6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1"
I20260619 18:59:05.640136 5879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:59:05.644263 5879 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4163a3cbf274424d90bfa4e55864ea6b in term 1.
I20260619 18:59:05.645198 5704 leader_election.cc:302] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee9dd3d05784bca89fbdf443f6f52d1, 4163a3cbf274424d90bfa4e55864ea6b; no voters:
I20260619 18:59:05.645737 5912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:59:05.666608 5912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [term 1 LEADER]: Becoming Leader. State: Replica: 4163a3cbf274424d90bfa4e55864ea6b, State: Running, Role: LEADER
I20260619 18:59:05.668618 5912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } }
I20260619 18:59:05.690593 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } attempt: 5
I20260619 18:59:05.701340 5936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4163a3cbf274424d90bfa4e55864ea6b. Latest consensus state: current_term: 1 leader_uuid: "4163a3cbf274424d90bfa4e55864ea6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } } }
I20260619 18:59:05.701807 5936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [sys.catalog]: This master's current role is: LEADER
Jun 19 18:59:05 dist-test-slave-dm3f krb5kdc[5675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
I20260619 18:59:05.713608 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.673528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.15.91.61:44539)
0619 18:59:05.673754 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.673760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.682246 (+ 8486us) server_negotiation.cc:407] Connection header received
0619 18:59:05.682569 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.682576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.682677 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.682782 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.685243 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.686354 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.694876 (+ 8522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.695640 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.695775 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.704001 (+ 8226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.704009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.704014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.704070 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.708293 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.709399 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.709411 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.709417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.709667 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.710262 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.710272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.710278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.710883 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.711266 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.712716 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.713007 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 18:59:05.716475 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.691284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:44497 (local address 127.0.0.1:44400)
0619 18:59:05.691692 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:05.691728 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:05.691843 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:05.692980 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:05.692989 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:05.693698 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:05.694000 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.694019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.695980 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.695990 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.697972 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:05.697999 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.699241 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.699248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:05.699807 (+ 559us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.701133 (+ 1326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:05.701201 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:05.706499 (+ 5298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:05.713143 (+ 6644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.713151 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.713157 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.713794 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.714316 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:05.714325 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:05.714331 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:05.714536 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:05.715327 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:05.715333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:05.715511 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 18:59:05.715678 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 18:59:05.716031 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
I20260619 18:59:05.717389 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.692067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.15.91.60:44497)
0619 18:59:05.692328 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.692337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.692368 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:59:05.692454 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.692461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.692566 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.692707 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.694290 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.695742 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.698258 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.699011 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.699159 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.706844 (+ 7685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.706852 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.706857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.706921 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.712967 (+ 6046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.713981 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.713989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.713994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.714164 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.714659 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.714667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.714671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.715093 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.715244 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.715912 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.716347 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:59:05.719014 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.644191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.15.91.62:39627)
0619 18:59:05.644418 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.644424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.666110 (+ 21686us) server_negotiation.cc:407] Connection header received
0619 18:59:05.667482 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.667490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.667585 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.667670 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.674382 (+ 6712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.675695 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.683895 (+ 8200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.684696 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.684838 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.695987 (+ 11149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.695995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.696001 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.696050 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.700206 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.713927 (+ 13721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.713934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.713937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.714137 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.715049 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.715055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.715059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.715495 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.715802 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.717647 (+ 1845us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.718400 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:59:05.725811 5829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5: Bootstrap starting.
I20260619 18:59:05.726794 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:05.644933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.15.91.60:44497)
0619 18:59:05.645150 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.645159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.670826 (+ 25667us) server_negotiation.cc:407] Connection header received
0619 18:59:05.671155 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.671163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.671277 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.671366 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.680525 (+ 9159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.681749 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.689453 (+ 7704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:05.690215 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:05.690344 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:05.697002 (+ 6658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:05.697013 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:05.697021 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:05.697077 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:05.709435 (+ 12358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.722640 (+ 13205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.722648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.722652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.722950 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:05.723977 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:05.723983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:05.723986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:05.724486 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:05.724747 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:05.725930 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0619 18:59:05.726354 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:59:05.732754 5829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5: Neither blocks nor log segments found. Creating new log.
I20260619 18:59:05.736191 5829 log.cc:824] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:59:05.736781 5940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:59:05.744890 5940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:59:05.749802 5829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5: No bootstrap required, opened a new log
W20260619 18:59:05.751111 5858 tablet.cc:2369] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:59:05.752352 5948 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:59:05.763329 5879 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [term 1 FOLLOWER]: Refusing update from remote peer 4163a3cbf274424d90bfa4e55864ea6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:59:05.764664 5936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:59:05.765676 5837 master_runner.cc:186] Error getting master registration for 127.15.91.61:44539: OK, instance_id { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" instance_seqno: 1781895544942483 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:59:05.780239 5837 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:59:05.783432 5829 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } }
I20260619 18:59:05.784075 5829 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb911dca77ae4249a22f089b406953b5, State: Initialized, Role: FOLLOWER
I20260619 18:59:05.784835 5829 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } }
I20260619 18:59:05.786440 5951 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } } }
I20260619 18:59:05.787448 5951 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:05.790771 5829 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:59:05.790861 5804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4163a3cbf274424d90bfa4e55864ea6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eb911dca77ae4249a22f089b406953b5"
I20260619 18:59:05.791234 5803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:59:05.795949 5803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [term 1 FOLLOWER]: Refusing update from remote peer 4163a3cbf274424d90bfa4e55864ea6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:59:05.800011 5804 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4163a3cbf274424d90bfa4e55864ea6b in term 1.
I20260619 18:59:05.800613 5926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4163a3cbf274424d90bfa4e55864ea6b. Latest consensus state: current_term: 1 leader_uuid: "4163a3cbf274424d90bfa4e55864ea6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } } }
I20260619 18:59:05.801046 5926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:05.802995 5936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:59:05.805575 5949 mvcc.cc:205] Tried to move back new op lower bound from 7298644155433828352 to 7298644155076067328. Current Snapshot: MvccSnapshot[applied={T|T < 7298644155433828352}]
I20260619 18:59:05.821028 5936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4163a3cbf274424d90bfa4e55864ea6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } } }
I20260619 18:59:05.821367 5936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [sys.catalog]: This master's current role is: LEADER
I20260619 18:59:05.822379 5940 catalog_manager.cc:1347] Generated new cluster ID: 755e7280cf41485ca4bb80ac360cd40f
I20260619 18:59:05.822503 5940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:59:05.824486 5926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4163a3cbf274424d90bfa4e55864ea6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } } }
I20260619 18:59:05.824769 5926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:05.839865 5951 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4163a3cbf274424d90bfa4e55864ea6b. Latest consensus state: current_term: 1 leader_uuid: "4163a3cbf274424d90bfa4e55864ea6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } } }
I20260619 18:59:05.840396 5951 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:05.845659 5912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4163a3cbf274424d90bfa4e55864ea6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } } }
I20260619 18:59:05.846021 5912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b [sys.catalog]: This master's current role is: LEADER
I20260619 18:59:05.851245 5940 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:59:05.852756 5940 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:59:05.867942 5951 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4163a3cbf274424d90bfa4e55864ea6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4163a3cbf274424d90bfa4e55864ea6b" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 39627 } } peers { permanent_uuid: "eb911dca77ae4249a22f089b406953b5" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 44539 } } peers { permanent_uuid: "3ee9dd3d05784bca89fbdf443f6f52d1" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 44497 } } }
I20260619 18:59:05.868253 5951 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:05.871996 5940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4163a3cbf274424d90bfa4e55864ea6b: Generated new TSK 0
I20260619 18:59:05.873260 5940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:59:05.890515 5969 catalog_manager.cc:1259] Loaded cluster ID: 755e7280cf41485ca4bb80ac360cd40f
I20260619 18:59:05.890769 5969 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5: loading cluster ID for follower catalog manager: success
I20260619 18:59:05.895406 5969 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5: acquiring CA information for follower catalog manager: success
I20260619 18:59:05.898691 5969 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb911dca77ae4249a22f089b406953b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 18:59:05.998754 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:05.985049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.15.91.60:44497)
0619 18:59:05.985355 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.985367 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:05.995614 (+ 10247us) server_negotiation.cc:407] Connection header received
0619 18:59:05.996295 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:05.996304 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:05.996433 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:05.996531 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:05.998302 (+ 1771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44404: BlockingRecv error: recv got EOF from 127.0.0.1:44404 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260619 18:59:06.002405 5909 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:05.985049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.15.91.62:39627)
0619 18:59:05.985613 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:05.985622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.000247 (+ 14625us) server_negotiation.cc:407] Connection header received
0619 18:59:06.000589 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.000596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.000692 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.000776 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.002098 (+ 1322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56036: BlockingRecv error: recv got EOF from 127.0.0.1:56036 (error 108)
Metrics: {"server-negotiator.queue_time_us":352}
W20260619 18:59:06.003536 5835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:06.001036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.15.91.61:44539)
0619 18:59:06.001257 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.001263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.001638 (+ 375us) server_negotiation.cc:407] Connection header received
0619 18:59:06.001906 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.001912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.002028 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.002105 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.003337 (+ 1232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53214: BlockingRecv error: recv got EOF from 127.0.0.1:53214 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:59:06.017547 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:06.009362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.15.91.61:44539)
0619 18:59:06.009615 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.009624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.011367 (+ 1743us) server_negotiation.cc:407] Connection header received
0619 18:59:06.011653 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.011658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.011816 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.012010 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:06.012479 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.013296 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.014810 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.015320 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.015487 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:06.016096 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:06.016735 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:06.016926 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:06.017018 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 18:59:06.017135 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 18:59:06.027536 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:06.024970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.15.91.60:44497)
0619 18:59:06.025224 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.025234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.025817 (+ 583us) server_negotiation.cc:407] Connection header received
0619 18:59:06.026089 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.026094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.026200 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.026326 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.027241 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44420: BlockingRecv error: recv got EOF from 127.0.0.1:44420 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:59:06.048722 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:06.042111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.15.91.61:44539)
0619 18:59:06.042431 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.042440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.043097 (+ 657us) server_negotiation.cc:407] Connection header received
0619 18:59:06.043344 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.043350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.043456 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.043577 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:06.044034 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.044837 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.046355 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.047037 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.047213 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:06.047742 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:06.047945 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:06.048049 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:06.048272 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 18:59:06.048365 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260619 18:59:06.058681 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:06.056292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.15.91.60:44497)
0619 18:59:06.056550 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.056560 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.057193 (+ 633us) server_negotiation.cc:407] Connection header received
0619 18:59:06.057449 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.057456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.057547 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.057665 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.058460 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44424: BlockingRecv error: recv got EOF from 127.0.0.1:44424 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 18:59:06.071295 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:06.065061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.15.91.61:44539)
0619 18:59:06.065305 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.065314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.065958 (+ 644us) server_negotiation.cc:407] Connection header received
0619 18:59:06.066190 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.066197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.066289 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.066399 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:06.066923 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.067738 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.069245 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.069751 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.069914 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:06.070382 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:06.070512 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:06.070623 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:06.070792 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 18:59:06.070887 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 18:59:06.080892 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:06.078395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.15.91.60:44497)
0619 18:59:06.078637 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.078647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.079299 (+ 652us) server_negotiation.cc:407] Connection header received
0619 18:59:06.079542 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.079548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.079652 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.079781 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.080607 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44428: BlockingRecv error: recv got EOF from 127.0.0.1:44428 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:59:06.095486 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:06.088688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.15.91.61:44539)
0619 18:59:06.089021 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.089031 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.089638 (+ 607us) server_negotiation.cc:407] Connection header received
0619 18:59:06.089889 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.089902 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.089978 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.090101 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:06.090591 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.091649 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.093325 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.093893 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.094070 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:06.094570 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:06.094697 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:06.094890 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:06.095049 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 18:59:06.095150 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
W20260619 18:59:06.106491 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:06.103711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.15.91.60:44497)
0619 18:59:06.104101 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.104111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.104834 (+ 723us) server_negotiation.cc:407] Connection header received
0619 18:59:06.105165 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.105172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.105268 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.105409 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.106234 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44442: BlockingRecv error: recv got EOF from 127.0.0.1:44442 (error 108)
Metrics: {"server-negotiator.queue_time_us":208}
I20260619 18:59:06.126170 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:06.119753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.15.91.61:44539)
0619 18:59:06.120024 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.120034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.120620 (+ 586us) server_negotiation.cc:407] Connection header received
0619 18:59:06.120861 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.120867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.120959 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.121072 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:06.121532 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.122366 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.124154 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.124656 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.124830 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:06.125287 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:06.125425 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:06.125540 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:06.125701 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 18:59:06.125796 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260619 18:59:06.142578 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:06.140291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.15.91.60:44497)
0619 18:59:06.140527 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.140537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.141096 (+ 559us) server_negotiation.cc:407] Connection header received
0619 18:59:06.141319 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.141325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.141412 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.141529 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.142330 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44444: BlockingRecv error: recv got EOF from 127.0.0.1:44444 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:59:06.171206 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:06.164875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.15.91.61:44539)
0619 18:59:06.165152 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.165181 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.165805 (+ 624us) server_negotiation.cc:407] Connection header received
0619 18:59:06.166059 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.166066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.166159 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.166272 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:06.166782 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.167644 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.169092 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.169579 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.169748 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:06.170272 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:06.170436 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:06.170552 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:06.170765 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 18:59:06.170871 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260619 18:59:06.187040 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:06.184294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.15.91.60:44497)
0619 18:59:06.184561 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.184570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.185190 (+ 620us) server_negotiation.cc:407] Connection header received
0619 18:59:06.185583 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.185590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.185688 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.185823 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.186802 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44450: BlockingRecv error: recv got EOF from 127.0.0.1:44450 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 18:59:06.228825 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:06.222762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.15.91.61:44539)
0619 18:59:06.223028 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.223038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.223628 (+ 590us) server_negotiation.cc:407] Connection header received
0619 18:59:06.223879 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.223884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.223961 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.224069 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:06.224522 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.225314 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.226747 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.227289 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.227449 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:06.227953 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:06.228087 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:06.228199 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:06.228359 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 18:59:06.228458 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 18:59:06.242949 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:06.240403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.15.91.60:44497)
0619 18:59:06.240671 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.240682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.241370 (+ 688us) server_negotiation.cc:407] Connection header received
0619 18:59:06.241619 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.241626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.241715 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.241842 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.242650 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44460: BlockingRecv error: recv got EOF from 127.0.0.1:44460 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:59:06.317844 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:06.310933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.15.91.61:44539)
0619 18:59:06.311158 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.311165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.311585 (+ 420us) server_negotiation.cc:407] Connection header received
0619 18:59:06.311751 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.311758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.311871 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.312009 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:06.312425 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.313603 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.315310 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.316108 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.316316 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:06.316897 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:06.317069 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:06.317200 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:06.317420 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 18:59:06.317523 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 18:59:06.327384 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:06.324944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.15.91.60:44497)
0619 18:59:06.325229 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.325236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.325615 (+ 379us) server_negotiation.cc:407] Connection header received
0619 18:59:06.325867 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.325871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.325954 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.326052 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.327139 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44470: BlockingRecv error: recv got EOF from 127.0.0.1:44470 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 18:59:06.468345 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:06.460336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.15.91.61:44539)
0619 18:59:06.460672 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.460680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.461169 (+ 489us) server_negotiation.cc:407] Connection header received
0619 18:59:06.461438 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.461443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.461519 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.461611 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:06.462304 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.463365 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.466135 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.466635 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.466884 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:06.467572 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:06.467699 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:06.467782 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:06.467974 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 18:59:06.468050 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260619 18:59:06.478374 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:06.475234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.15.91.60:44497)
0619 18:59:06.475504 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.475514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.476059 (+ 545us) server_negotiation.cc:407] Connection header received
0619 18:59:06.476530 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.476544 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.476663 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.476800 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.478058 (+ 1258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44478: BlockingRecv error: recv got EOF from 127.0.0.1:44478 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:59:06.747215 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:06.741252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.15.91.61:44539)
0619 18:59:06.741538 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.741544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.741729 (+ 185us) server_negotiation.cc:407] Connection header received
0619 18:59:06.741933 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.741938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.742015 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.742109 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:06.742565 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.743363 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.744881 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:06.745324 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:06.745455 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:06.746083 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:06.746291 (+ 208us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:06.746576 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:06.746708 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 18:59:06.746865 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260619 18:59:06.756273 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:06.753884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.15.91.60:44497)
0619 18:59:06.754136 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:06.754144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:06.754585 (+ 441us) server_negotiation.cc:407] Connection header received
0619 18:59:06.754868 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:06.754875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:06.754963 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:06.755086 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:06.755977 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44494: BlockingRecv error: recv got EOF from 127.0.0.1:44494 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 18:59:06.759995 5948 catalog_manager.cc:1259] Loaded cluster ID: 755e7280cf41485ca4bb80ac360cd40f
I20260619 18:59:06.760155 5948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1: loading cluster ID for follower catalog manager: success
I20260619 18:59:06.764655 5948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1: acquiring CA information for follower catalog manager: success
I20260619 18:59:06.767792 5948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ee9dd3d05784bca89fbdf443f6f52d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:59:07.278144 6085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:07.272049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.15.91.61:44539)
0619 18:59:07.272515 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:07.272520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:07.273046 (+ 526us) server_negotiation.cc:407] Connection header received
0619 18:59:07.273187 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:07.273192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:07.273277 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:07.273390 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:07.273952 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:07.274780 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:07.276278 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:07.276740 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:07.276876 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:07.277345 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:07.277468 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:07.277544 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:07.277702 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 18:59:07.277778 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":257,"threads_started":1}
I20260619 18:59:07.290895 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:07.284303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.15.91.60:44497)
0619 18:59:07.284868 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:07.284875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:07.285009 (+ 134us) server_negotiation.cc:407] Connection header received
0619 18:59:07.285266 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:07.285271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:07.285443 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:07.285591 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:07.286093 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:07.286897 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:07.288282 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:07.288742 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:07.288912 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:07.289520 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:07.290105 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:07.290287 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:07.290427 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 18:59:07.290529 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":266,"threads_started":1}
I20260619 18:59:07.308291 6085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:07.299814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.15.91.61:44539)
0619 18:59:07.300063 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:07.300069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:07.301125 (+ 1056us) server_negotiation.cc:407] Connection header received
0619 18:59:07.301359 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:07.301365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:07.301470 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:07.301546 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:07.302371 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:07.303421 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:07.306016 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:07.306545 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:07.306689 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:07.307400 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:07.307528 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:07.307644 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:07.307849 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 18:59:07.307945 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 18:59:07.308629 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:07.297705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.15.91.62:39627)
0619 18:59:07.300750 (+ 3045us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:07.300757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:07.300781 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:59:07.300934 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:07.300940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:07.301234 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:07.301479 (+ 245us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:07.302284 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:07.303366 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:07.305629 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:07.306253 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:07.306414 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:07.307135 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:07.307724 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:07.307851 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:07.308036 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 18:59:07.308132 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2848,"thread_start_us":2717,"threads_started":1}
I20260619 18:59:07.309127 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:07.297705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.15.91.60:44497)
0619 18:59:07.297949 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:07.297959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:07.300548 (+ 2589us) server_negotiation.cc:407] Connection header received
0619 18:59:07.300938 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:07.300943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:07.301041 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:07.301162 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:07.301760 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:07.303003 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:07.305896 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:07.306581 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:07.306774 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:07.307415 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:07.307573 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:07.307667 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:07.308279 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 18:59:07.308369 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:59:07.315920 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5688
I20260619 18:59:07.317239 5750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:59:07.433790 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 5688
W20260619 18:59:07.455833 5780 connection.cc:535] server connection from 127.0.0.1:53174 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:59:07.455961 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5762
I20260619 18:59:07.457123 5824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:59:07.567791 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 5762
I20260619 18:59:07.589419 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 5837
I20260619 18:59:07.590559 5899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:59:07.700343 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 5837
2026-06-19T18:59:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5204 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:59:07 dist-test-slave-dm3f krb5kdc[6106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:59:07 dist-test-slave-dm3f krb5kdc[6106](info): set up 2 sockets
Jun 19 18:59:07 dist-test-slave-dm3f krb5kdc[6106](info): commencing operation
krb5kdc: starting...
W20260619 18:59:08.827494 15724 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/test-user.keytab.
Jun 19 18:59:09 dist-test-slave-dm3f krb5kdc[6106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:59:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.91.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
Entry for principal kudu/127.15.91.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab.
I20260619 18:59:09.106691 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46809
--webserver_interface=127.15.91.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46809,127.15.91.61:35793,127.15.91.60:35269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:59:09.480141 6119 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:59:09.480576 6119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:59:09.480722 6119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:59:09.480862 6119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:59:09.489151 6119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:59:09.489275 6119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:59:09.489378 6119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:59:09.489470 6119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:59:09.489560 6119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:59:09.500581 6119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46809,127.15.91.61:35793,127.15.91.60:35269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.62.keytab
--principal=kudu/127.15.91.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.62:46809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.91.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:59:09.502177 6119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:59:09.503793 6119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:59:09.517058 6124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:09.517021 6125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:09.517025 6127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:59:09.518505 6119 server_base.cc:1034] running on GCE node
I20260619 18:59:09.519467 6119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:59:09.521224 6119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:59:09.522547 6119 hybrid_clock.cc:648] HybridClock initialized: now 1781895549522467 us; error 57 us; skew 500 ppm
I20260619 18:59:09.523025 6119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:59:09 dist-test-slave-dm3f krb5kdc[6106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895549, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:59:09.529865 6119 init.cc:377] Logged in from keytab as kudu/127.15.91.62@KRBTEST.COM (short username kudu)
I20260619 18:59:09.531847 6119 webserver.cc:460] Webserver started at http://127.15.91.62:45691/ using document root <none> and password file <none>
I20260619 18:59:09.532730 6119 fs_manager.cc:359] Metadata directory not provided
I20260619 18:59:09.532908 6119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:59:09.533278 6119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:59:09.536955 6119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/data/instance:
uuid: "e9477a6d594443279482c09082c16af7"
format_stamp: "Formatted at 2026-06-19 18:59:09 on dist-test-slave-dm3f"
server_key: "8cb8f49590b28aa7b11b66a022826b7a"
server_key_iv: "c6249cd61e27bbf1e1993ff10c013cfd"
server_key_version: "encryptionkey@0"
I20260619 18:59:09.538012 6119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/wal/instance:
uuid: "e9477a6d594443279482c09082c16af7"
format_stamp: "Formatted at 2026-06-19 18:59:09 on dist-test-slave-dm3f"
server_key: "8cb8f49590b28aa7b11b66a022826b7a"
server_key_iv: "c6249cd61e27bbf1e1993ff10c013cfd"
server_key_version: "encryptionkey@0"
I20260619 18:59:09.544515 6119 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260619 18:59:09.549162 6134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:59:09.551115 6119 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:59:09.551379 6119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/wal
uuid: "e9477a6d594443279482c09082c16af7"
format_stamp: "Formatted at 2026-06-19 18:59:09 on dist-test-slave-dm3f"
server_key: "8cb8f49590b28aa7b11b66a022826b7a"
server_key_iv: "c6249cd61e27bbf1e1993ff10c013cfd"
server_key_version: "encryptionkey@0"
I20260619 18:59:09.551656 6119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:59:09.567044 6119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:59:09.571928 6119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:59:09.572252 6119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:59:09.593673 6119 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.62:46809
I20260619 18:59:09.593691 6185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.62:46809 every 8 connection(s)
I20260619 18:59:09.595310 6119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/data/info.pb
I20260619 18:59:09.596639 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 6119
I20260619 18:59:09.596876 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-0/wal/instance
I20260619 18:59:09.597357 15724 external_mini_cluster.cc:1428] Setting key a692debfba98a08d9b314c8a08a84150
I20260619 18:59:09.600034 6186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:59:09.607403 6186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.91.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.61@KRBTEST.COM" created.
Jun 19 18:59:09 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895549, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
I20260619 18:59:09.635071 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:09.611099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.15.91.62:46809)
0619 18:59:09.613014 (+ 1915us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:09.613024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:09.613087 (+ 63us) server_negotiation.cc:407] Connection header received
0619 18:59:09.614895 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:09.614902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:09.615196 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:09.615449 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:09.616860 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:09.618140 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:09.619787 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:09.620409 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:09.620641 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:09.625621 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:09.625638 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:09.625662 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:09.625716 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:09.630089 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:09.631097 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:09.631110 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:09.631116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:09.631366 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:09.631816 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:09.631824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:09.631830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:09.632661 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:09.632917 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:09.633510 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 18:59:09.634000 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":1688,"thread_start_us":1433,"threads_started":1}
I20260619 18:59:09.635142 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:09.611081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46809 (local address 127.0.0.1:44032)
0619 18:59:09.612086 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:09.612138 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:09.614369 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:09.615456 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:09.615462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:09.616260 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:09.616610 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:09.616630 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:09.618331 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:09.618345 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:09.619606 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:09.619620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:09.620680 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:09.620686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:09.620881 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:09.621627 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:09.621656 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:09.625263 (+ 3607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:09.630314 (+ 5051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:09.630335 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:09.630347 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:09.630905 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:09.631499 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:09.631502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:09.631505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:09.631680 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:09.632827 (+ 1147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:09.632833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:09.633042 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 18:59:09.633280 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 18:59:09.633754 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":423,"threads_started":1}
I20260619 18:59:09.638571 6186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:59:09.639909 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:09.639449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60770)
0619 18:59:09.639652 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:09.639777 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 18:59:09.640198 6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
I20260619 18:59:09.641971 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:09.641535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60778)
0619 18:59:09.641739 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:09.641814 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 18:59:09.642225 6186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:35793: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab.
Entry for principal kudu/127.15.91.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab.
I20260619 18:59:09.659827 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.61:35793
--webserver_interface=127.15.91.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab
--principal=kudu/127.15.91.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46809,127.15.91.61:35793,127.15.91.60:35269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:59:09.691605 6186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } attempt: 1
I20260619 18:59:09.692893 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:09.692422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60786)
0619 18:59:09.692637 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:09.692733 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 18:59:09.694053 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:09.693638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60800)
0619 18:59:09.693814 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:09.693895 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 18:59:09.694267 6186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:35793: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
I20260619 18:59:09.762555 6186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } attempt: 2
I20260619 18:59:09.763862 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:09.763330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60804)
0619 18:59:09.763618 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:09.763711 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 18:59:09.765080 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:09.764640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60816)
0619 18:59:09.764821 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:09.764911 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 18:59:09.765339 6186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:35793: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
I20260619 18:59:09.856681 6186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } attempt: 3
I20260619 18:59:09.857926 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:09.857463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60818)
0619 18:59:09.857673 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:09.857777 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 18:59:09.859236 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:09.858797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60828)
0619 18:59:09.858965 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:09.859051 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 18:59:09.859550 6186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:35793: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
I20260619 18:59:10.002863 6186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } attempt: 4
I20260619 18:59:10.004375 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.003715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60840)
0619 18:59:10.004089 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.004207 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260619 18:59:10.005651 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.005206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60842)
0619 18:59:10.005397 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.005500 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 18:59:10.005955 6186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.61:35793: Network error: Client connection negotiation failed: client connection to 127.15.91.61:35793: connect: Connection refused (error 111)
W20260619 18:59:10.025197 6193 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:59:10.025799 6193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:59:10.025957 6193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:59:10.026103 6193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:59:10.034149 6193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:59:10.034328 6193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:59:10.034435 6193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:59:10.034524 6193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:59:10.034632 6193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:59:10.045573 6193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46809,127.15.91.61:35793,127.15.91.60:35269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.61.keytab
--principal=kudu/127.15.91.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.61:35793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.91.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:59:10.047094 6193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:59:10.048568 6193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:59:10.062489 6198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:10.062584 6201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:10.062565 6199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:59:10.063688 6193 server_base.cc:1034] running on GCE node
I20260619 18:59:10.064683 6193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:59:10.066356 6193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:59:10.067669 6193 hybrid_clock.cc:648] HybridClock initialized: now 1781895550067556 us; error 90 us; skew 500 ppm
I20260619 18:59:10.068076 6193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895550, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:59:10.071811 6193 init.cc:377] Logged in from keytab as kudu/127.15.91.61@KRBTEST.COM (short username kudu)
I20260619 18:59:10.073585 6193 webserver.cc:460] Webserver started at http://127.15.91.61:41199/ using document root <none> and password file <none>
I20260619 18:59:10.074424 6193 fs_manager.cc:359] Metadata directory not provided
I20260619 18:59:10.074601 6193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:59:10.075027 6193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:59:10.078455 6193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/data/instance:
uuid: "410b2dcffa8046918bae9b1277bf5b40"
format_stamp: "Formatted at 2026-06-19 18:59:10 on dist-test-slave-dm3f"
server_key: "c22366028a3f2a949353e118d79f1258"
server_key_iv: "d9b0acd6c09a7df0ca3683a8ec35a9c9"
server_key_version: "encryptionkey@0"
I20260619 18:59:10.079512 6193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/wal/instance:
uuid: "410b2dcffa8046918bae9b1277bf5b40"
format_stamp: "Formatted at 2026-06-19 18:59:10 on dist-test-slave-dm3f"
server_key: "c22366028a3f2a949353e118d79f1258"
server_key_iv: "d9b0acd6c09a7df0ca3683a8ec35a9c9"
server_key_version: "encryptionkey@0"
I20260619 18:59:10.085682 6193 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:59:10.090193 6208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:59:10.091876 6193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:59:10.092132 6193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/wal
uuid: "410b2dcffa8046918bae9b1277bf5b40"
format_stamp: "Formatted at 2026-06-19 18:59:10 on dist-test-slave-dm3f"
server_key: "c22366028a3f2a949353e118d79f1258"
server_key_iv: "d9b0acd6c09a7df0ca3683a8ec35a9c9"
server_key_version: "encryptionkey@0"
I20260619 18:59:10.092428 6193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:59:10.110222 6193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:59:10.115017 6193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:59:10.115334 6193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:59:10.136512 6193 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.61:35793
I20260619 18:59:10.136559 6259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.61:35793 every 8 connection(s)
I20260619 18:59:10.138115 6193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/data/info.pb
I20260619 18:59:10.142407 6260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:59:10.148579 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 6193
I20260619 18:59:10.148802 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-1/wal/instance
I20260619 18:59:10.149348 15724 external_mini_cluster.cc:1428] Setting key e8094c28a01500beb979cb32fdb53872
I20260619 18:59:10.152693 6260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895550, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.91.60@KRBTEST.COM; defaulting to no policy
I20260619 18:59:10.180768 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.156770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.15.91.62:46809)
0619 18:59:10.157458 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.157466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.158814 (+ 1348us) server_negotiation.cc:407] Connection header received
0619 18:59:10.161536 (+ 2722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.161545 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.161663 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.161814 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.163817 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.165117 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.167164 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.167970 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.168152 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.172425 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.172436 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.172444 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.172508 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.176819 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.177962 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.177969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.177973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.178159 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.178824 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.178828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.178831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.179160 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.179292 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.180253 (+ 961us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.180475 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":397,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.91.60@KRBTEST.COM" created.
I20260619 18:59:10.182353 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.157572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46809 (local address 127.0.0.1:44040)
0619 18:59:10.158553 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.158615 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:10.161149 (+ 2534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:10.162068 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:10.162078 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:10.162838 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:10.163507 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.163528 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.165389 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.165395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.166920 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.166943 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.168264 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.168269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.168535 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.169423 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:10.169452 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:10.172022 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:10.177100 (+ 5078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.177109 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.177127 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.177733 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.178314 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.178317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.178320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.178614 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.179341 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:10.179348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:10.179838 (+ 490us) client_negotiation.cc:769] Sending connection context
0619 18:59:10.180149 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 18:59:10.180741 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":374,"threads_started":1}
I20260619 18:59:10.184619 6260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895550, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:59:10.201885 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.185607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60856)
0619 18:59:10.185816 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.185835 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:10.185977 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:10.187273 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:10.187277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:10.187806 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:10.187946 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.187956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.189550 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.189558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.190449 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.190463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.191490 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.191496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.191697 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.192385 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:10.192406 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:10.195008 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:10.199013 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.199018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.199023 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.199421 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.199858 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.199863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.199867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.200011 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.200662 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:10.200666 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:10.200887 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 18:59:10.201037 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 18:59:10.201386 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":1}
I20260619 18:59:10.201952 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.185940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.15.91.61:35793)
0619 18:59:10.186685 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.186691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.186784 (+ 93us) server_negotiation.cc:407] Connection header received
0619 18:59:10.186866 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.186873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.187090 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.187288 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.188151 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.189380 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.190625 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.191315 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.191474 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.195282 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.195300 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.195316 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.195350 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.198834 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.199583 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.199599 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.199601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.199726 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.200131 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.200142 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.200145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.200515 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.200767 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.201223 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.201508 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":581,"thread_start_us":432,"threads_started":1}
I20260619 18:59:10.204704 6260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:59:10.206151 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.205638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54486)
0619 18:59:10.205847 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.205976 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 18:59:10.206470 6212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
I20260619 18:59:10.208456 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.208027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54498)
0619 18:59:10.208207 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.208310 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 18:59:10.208840 6260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:35269: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.91.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab.
Entry for principal kudu/127.15.91.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab.
I20260619 18:59:10.216002 15724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskualRwQ/build/asan/bin/kudu
/tmp/dist-test-taskualRwQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.60:35269
--webserver_interface=127.15.91.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab
--principal=kudu/127.15.91.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.91.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.91.62:46809,127.15.91.61:35793,127.15.91.60:35269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:59:10.258210 6260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } attempt: 1
I20260619 18:59:10.259626 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.259062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54506)
0619 18:59:10.259293 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.259427 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 18:59:10.261027 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.260552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54518)
0619 18:59:10.260755 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.260868 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 18:59:10.261416 6260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:35269: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
I20260619 18:59:10.305265 6186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } attempt: 5
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895549, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:59:10.322165 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.306081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60860)
0619 18:59:10.306346 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.306371 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:10.306538 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:10.307029 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:10.307034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:10.307612 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:10.307878 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.307893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.308997 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.309004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.310851 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.310906 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.311858 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.311866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.312177 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.312900 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:10.312926 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:10.315638 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:10.319197 (+ 3559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.319204 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.319209 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.319665 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.320142 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.320147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.320150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.320324 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.320963 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:10.320968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:10.321200 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 18:59:10.321426 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 18:59:10.321788 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 18:59:10.322165 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.306219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.15.91.61:35793)
0619 18:59:10.306413 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.306419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.306440 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:59:10.306755 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.306765 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.306857 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.306949 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.308076 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.308789 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.311122 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.311611 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.311737 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.315934 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.315939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.315943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.315985 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.318984 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.319833 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.319842 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.319852 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.319998 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.320461 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.320474 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.320476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.320812 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.320998 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.321560 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.321815 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:59:10.323594 6186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:59:10.324930 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.324451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54524)
0619 18:59:10.324652 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.324759 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 18:59:10.326118 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.325718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54536)
0619 18:59:10.325887 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.325978 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 18:59:10.326386 6186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:35269: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
I20260619 18:59:10.329690 6260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } attempt: 2
I20260619 18:59:10.331144 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.330507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54544)
0619 18:59:10.330793 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.330912 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 18:59:10.332480 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.332004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54550)
0619 18:59:10.332192 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.332305 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 18:59:10.332747 6260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:35269: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
I20260619 18:59:10.377660 6186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } attempt: 1
I20260619 18:59:10.379067 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.378408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54552)
0619 18:59:10.378670 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.378836 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260619 18:59:10.380398 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.379892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54568)
0619 18:59:10.380114 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.380226 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 18:59:10.380689 6186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:35269: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
I20260619 18:59:10.424022 6260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } attempt: 3
I20260619 18:59:10.425523 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.424806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54572)
0619 18:59:10.425109 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.425254 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 18:59:10.426980 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.426407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54580)
0619 18:59:10.426669 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.426805 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260619 18:59:10.427273 6260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:35269: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
I20260619 18:59:10.448976 6186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } attempt: 2
I20260619 18:59:10.450460 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.449786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54584)
0619 18:59:10.450080 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.450216 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 18:59:10.451786 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.451266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54590)
0619 18:59:10.451471 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.451585 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 18:59:10.452054 6186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:35269: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
I20260619 18:59:10.558321 6186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } attempt: 3
I20260619 18:59:10.559731 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.559113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54598)
0619 18:59:10.559380 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.559503 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 18:59:10.561108 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.560563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54604)
0619 18:59:10.560786 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.560923 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260619 18:59:10.561371 6186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:35269: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
I20260619 18:59:10.570567 6260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } attempt: 4
I20260619 18:59:10.572072 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.571388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54606)
0619 18:59:10.571694 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.571827 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 18:59:10.573467 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.572988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54616)
0619 18:59:10.573173 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.573289 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 18:59:10.573764 6260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.91.60:35269: Network error: Client connection negotiation failed: client connection to 127.15.91.60:35269: connect: Connection refused (error 111)
W20260619 18:59:10.599313 6268 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:59:10.599689 6268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:59:10.599819 6268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:59:10.599953 6268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:59:10.607969 6268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:59:10.608074 6268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:59:10.608176 6268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:59:10.608276 6268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:59:10.608388 6268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:59:10.619556 6268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.91.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.91.62:46809,127.15.91.61:35793,127.15.91.60:35269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/krb5kdc/kudu_127.15.91.60.keytab
--principal=kudu/127.15.91.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.91.60:35269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.91.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:59:10.621017 6268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:59:10.622511 6268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:59:10.635942 6273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:10.636053 6276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:59:10.636314 6274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:59:10.636861 6268 server_base.cc:1034] running on GCE node
I20260619 18:59:10.637794 6268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:59:10.639564 6268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:59:10.640851 6268 hybrid_clock.cc:648] HybridClock initialized: now 1781895550640770 us; error 50 us; skew 500 ppm
I20260619 18:59:10.641271 6268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895550, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:59:10.644922 6268 init.cc:377] Logged in from keytab as kudu/127.15.91.60@KRBTEST.COM (short username kudu)
I20260619 18:59:10.646782 6268 webserver.cc:460] Webserver started at http://127.15.91.60:34991/ using document root <none> and password file <none>
I20260619 18:59:10.647622 6268 fs_manager.cc:359] Metadata directory not provided
I20260619 18:59:10.647781 6268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:59:10.648149 6268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:59:10.651618 6268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/data/instance:
uuid: "470638aac7b049b88e8b74bf1b24d83e"
format_stamp: "Formatted at 2026-06-19 18:59:10 on dist-test-slave-dm3f"
server_key: "5b0c93522cc7c2be4bac7d1a153fd523"
server_key_iv: "49086ed7a95d4072c5b9839e3e22b7ea"
server_key_version: "encryptionkey@0"
I20260619 18:59:10.652594 6268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/wal/instance:
uuid: "470638aac7b049b88e8b74bf1b24d83e"
format_stamp: "Formatted at 2026-06-19 18:59:10 on dist-test-slave-dm3f"
server_key: "5b0c93522cc7c2be4bac7d1a153fd523"
server_key_iv: "49086ed7a95d4072c5b9839e3e22b7ea"
server_key_version: "encryptionkey@0"
I20260619 18:59:10.659056 6268 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:59:10.663383 6283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:59:10.665076 6268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:59:10.665433 6268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/data,/tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/wal
uuid: "470638aac7b049b88e8b74bf1b24d83e"
format_stamp: "Formatted at 2026-06-19 18:59:10 on dist-test-slave-dm3f"
server_key: "5b0c93522cc7c2be4bac7d1a153fd523"
server_key_iv: "49086ed7a95d4072c5b9839e3e22b7ea"
server_key_version: "encryptionkey@0"
I20260619 18:59:10.665710 6268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:59:10.688694 6268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:59:10.693600 6268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:59:10.693953 6268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:59:10.715992 6268 rpc_server.cc:307] RPC server started. Bound to: 127.15.91.60:35269
I20260619 18:59:10.716017 6334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.91.60:35269 every 8 connection(s)
I20260619 18:59:10.717833 6268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/data/info.pb
I20260619 18:59:10.719884 15724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskualRwQ/build/asan/bin/kudu as pid 6268
I20260619 18:59:10.720090 15724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskualRwQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781895164554191-15724-0/minicluster-data/master-2/wal/instance
I20260619 18:59:10.720687 15724 external_mini_cluster.cc:1428] Setting key 7126b97806ede894618657303f15ff09
I20260619 18:59:10.723472 6335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:59:10.733186 6335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
I20260619 18:59:10.738664 6186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } attempt: 4
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895550, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895549, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.62@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
I20260619 18:59:10.751503 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.722972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.15.91.60:35269)
0619 18:59:10.723998 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.724013 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.724063 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:59:10.726064 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.726094 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.726689 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.727264 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.729363 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.731174 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.733113 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.734118 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.734402 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.741004 (+ 6602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.741022 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.741037 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.741076 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.745512 (+ 4436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.746912 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.746929 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.746938 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.747126 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.747628 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.747634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.747637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.748376 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.748754 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.749316 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.749843 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":491,"threads_started":1}
I20260619 18:59:10.757977 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.736791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.62:46809 (local address 127.0.0.1:44044)
0619 18:59:10.737889 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.737943 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:10.738141 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:10.738671 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:10.738679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:10.739768 (+ 1089us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:10.740196 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.740209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.741670 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.741682 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.743769 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.743782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.744906 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.744912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.745077 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.745845 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:10.745881 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:10.750015 (+ 4134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:10.754063 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.754075 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.754082 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.754764 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.755264 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.755270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.755275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.755488 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.756490 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:10.756500 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:10.756758 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 18:59:10.756984 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 18:59:10.757445 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":1,"thread_start_us":629,"threads_started":1}
I20260619 18:59:10.758172 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.736791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.15.91.62:46809)
0619 18:59:10.737514 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.737523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.738064 (+ 541us) server_negotiation.cc:407] Connection header received
0619 18:59:10.738309 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.738315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.738408 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.738543 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.740380 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.741441 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.743954 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.744475 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.744650 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.750309 (+ 5659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.750322 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.750327 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.750377 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.753894 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.754931 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.754936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.754938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.755079 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.755664 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.755673 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.755679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.756310 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.756516 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.757215 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.757565 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":305,"threads_started":1}
I20260619 18:59:10.759601 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.739370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54628)
0619 18:59:10.739581 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.739602 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:10.739721 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:10.740798 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:10.740804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:10.741302 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:10.741540 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.741554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.743626 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.743636 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.745326 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.745345 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.746954 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.746962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.747201 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.747921 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:10.747941 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:10.750866 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:10.756528 (+ 5662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.756533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.756536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.757142 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.757655 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.757661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.757665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.757843 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.758548 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:10.758554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:10.758799 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 18:59:10.758921 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 18:59:10.759177 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":1}
I20260619 18:59:10.760828 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.739584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.15.91.60:35269)
0619 18:59:10.740419 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.740427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.740453 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:59:10.740509 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.740514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.740586 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.740716 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.741716 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.743357 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.745518 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.746390 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.746564 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.751162 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.751172 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.751177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.751219 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.756348 (+ 5129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.757345 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.757352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.757359 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.757508 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.757991 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.757999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.758005 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.758386 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.759915 (+ 1529us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.760064 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.760345 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":679,"thread_start_us":436,"threads_started":1}
I20260619 18:59:10.764209 6335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:59:10.768447 6186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7: Bootstrap starting.
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.62@KRBTEST.COM
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895550, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:59:10.773707 6186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7: Neither blocks nor log segments found. Creating new log.
I20260619 18:59:10.775257 6186 log.cc:824] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:59:10.780371 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.765024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.61:35793 (local address 127.0.0.1:60868)
0619 18:59:10.765315 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.765336 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:10.765463 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:10.765820 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:10.765825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:10.766260 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:10.766419 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.766428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.767523 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.767529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.768431 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.768444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.769335 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.769342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.769538 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.770198 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:10.770223 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:10.773115 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:10.777125 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.777130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.777134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.777700 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.778297 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.778303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.778306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.778514 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.779309 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:10.779314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:10.779480 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 18:59:10.779640 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 18:59:10.779973 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260619 18:59:10.780405 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.765153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.15.91.61:35793)
0619 18:59:10.765326 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.765334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.765389 (+ 55us) server_negotiation.cc:407] Connection header received
0619 18:59:10.765602 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.765609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.765679 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.765791 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.766581 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.767369 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.768604 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.769079 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.769236 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.773423 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.773432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.773440 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.773498 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.776884 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.777857 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.777865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.777870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.778079 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.778689 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.778694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.778697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.779144 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.779343 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.779750 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.780002 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:59:10.780830 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.755388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44048 (local address 127.15.91.62:46809)
0619 18:59:10.758505 (+ 3117us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.758513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.758537 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:59:10.759052 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.759060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.759206 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.759363 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.761741 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.763201 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.765162 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.765844 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.765973 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.771772 (+ 5799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.771777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.771781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.771821 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.775715 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.777041 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.777051 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.777054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.777281 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.777932 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.777942 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.777948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.778528 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.778889 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.780173 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.780471 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2967,"thread_start_us":7525,"threads_started":1}
I20260619 18:59:10.781859 6186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7: No bootstrap required, opened a new log
I20260619 18:59:10.781852 6335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895550, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.60@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.91.61@KRBTEST.COM
I20260619 18:59:10.794483 6186 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } }
I20260619 18:59:10.795027 6186 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9477a6d594443279482c09082c16af7, State: Initialized, Role: FOLLOWER
I20260619 18:59:10.795657 6186 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } }
I20260619 18:59:10.796921 6344 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } } }
I20260619 18:59:10.797562 6344 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:10.798516 6186 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:59:10.799433 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.782677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.15.91.60:35269)
0619 18:59:10.782939 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.782944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.783168 (+ 224us) server_negotiation.cc:407] Connection header received
0619 18:59:10.783208 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.783212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.783288 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.783377 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.784461 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.785534 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.787268 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.787919 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.788025 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.792540 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.792545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.792549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.792582 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.795903 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.797046 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.797053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.797056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.797246 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.797933 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.797944 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.797947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.798241 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.798411 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.798837 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.799109 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260619 18:59:10.800546 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.782677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54630)
0619 18:59:10.782910 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.782935 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:10.783067 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:10.783536 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:10.783541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:10.784090 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:10.784307 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.784322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.785699 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.785705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.787100 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.787118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.788290 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.788299 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.788520 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.789167 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:10.789194 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:10.792303 (+ 3109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:10.796232 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.796240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.796246 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.796893 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.797397 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.797404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.797408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.797804 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.798365 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:10.798371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:10.798524 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 18:59:10.799721 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0619 18:59:10.800126 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 18:59:10.801698 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.783292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.15.91.61:35793)
0619 18:59:10.783459 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.783464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.783605 (+ 141us) server_negotiation.cc:407] Connection header received
0619 18:59:10.783800 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.783805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.783880 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.783984 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.785234 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.785982 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.787342 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.787812 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.787970 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.793506 (+ 5536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.793513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.793517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.793564 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.797398 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.798502 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.798509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.798514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.798709 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.799247 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.799254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.799257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.799651 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.799994 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.800934 (+ 940us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.801381 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:59:10.807070 15724 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:59:10.808560 6335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e: Bootstrap starting.
I20260619 18:59:10.809612 6161 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e9477a6d594443279482c09082c16af7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44048
I20260619 18:59:10.810068 6161 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 18:59:10.810293 6161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:59:10.813005 6161 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } }
I20260619 18:59:10.814867 6335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e: Neither blocks nor log segments found. Creating new log.
I20260619 18:59:10.815689 6161 leader_election.cc:288] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [CANDIDATE]: Term 1 election: Requested vote from peers 410b2dcffa8046918bae9b1277bf5b40 (127.15.91.61:35793), 470638aac7b049b88e8b74bf1b24d83e (127.15.91.60:35269)
I20260619 18:59:10.816838 6335 log.cc:824] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e: Log is configured to *not* fsync() on all Append() calls
W20260619 18:59:10.825467 6140 tablet.cc:2369] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:59:10.826504 6335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e: No bootstrap required, opened a new log
W20260619 18:59:10.836367 6357 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:59:10.836539 6357 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:59:10.844245 6335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } }
I20260619 18:59:10.844798 6335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470638aac7b049b88e8b74bf1b24d83e, State: Initialized, Role: FOLLOWER
I20260619 18:59:10.845436 6335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } }
I20260619 18:59:10.830202 6119 master_runner.cc:186] Error getting master registration for 127.15.91.61:35793: OK, instance_id { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" instance_seqno: 1781895550130197 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:59:10.848011 6119 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:59:10.852651 6365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } } }
I20260619 18:59:10.853133 6365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:10.855348 6335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:59:10.856355 6310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9477a6d594443279482c09082c16af7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "470638aac7b049b88e8b74bf1b24d83e"
I20260619 18:59:10.857105 6310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:59:10.861150 6310 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9477a6d594443279482c09082c16af7 in term 1.
I20260619 18:59:10.862504 6138 leader_election.cc:302] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 470638aac7b049b88e8b74bf1b24d83e, e9477a6d594443279482c09082c16af7; no voters:
I20260619 18:59:10.863194 6344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:59:10.870088 6344 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [term 1 LEADER]: Becoming Leader. State: Replica: e9477a6d594443279482c09082c16af7, State: Running, Role: LEADER
I20260619 18:59:10.872999 6260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } attempt: 5
I20260619 18:59:10.874004 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.826612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.15.91.61:35793)
0619 18:59:10.826844 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.826850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.832807 (+ 5957us) server_negotiation.cc:407] Connection header received
0619 18:59:10.833142 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.833149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.833242 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.833356 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.850078 (+ 16722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.851669 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.853867 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.854555 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.854685 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.861359 (+ 6674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.861368 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.861372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.861429 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.865940 (+ 4511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.870565 (+ 4625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.870576 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.870583 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.870898 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.871633 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.871643 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.871648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.872126 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.872452 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.873314 (+ 862us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.873606 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:59:10.872972 6344 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } }
Jun 19 18:59:10 dist-test-slave-dm3f krb5kdc[6106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895550, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.91.61@KRBTEST.COM for kudu/127.15.91.60@KRBTEST.COM
I20260619 18:59:10.885883 6377 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:59:10.890642 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.830387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.15.91.60:35269)
0619 18:59:10.830601 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.830612 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.835405 (+ 4793us) server_negotiation.cc:407] Connection header received
0619 18:59:10.836173 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.836181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.836281 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.836375 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.854746 (+ 18371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.856056 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.861002 (+ 4946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.861782 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.861897 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.871749 (+ 9852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.871761 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.871769 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.871848 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.884069 (+ 12221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.885933 (+ 1864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.885939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.885943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.886493 (+ 550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.888022 (+ 1529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.888029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.888033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.888489 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.888828 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.889964 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.890225 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:59:10.893213 6366 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9477a6d594443279482c09082c16af7. Latest consensus state: current_term: 1 leader_uuid: "e9477a6d594443279482c09082c16af7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } } }
I20260619 18:59:10.893718 6366 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:59:10.896370 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.873872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.15.91.60:35269)
0619 18:59:10.874082 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.874089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.874455 (+ 366us) server_negotiation.cc:407] Connection header received
0619 18:59:10.874545 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.874552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.874657 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.874919 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.876299 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.877238 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.879281 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.880108 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.880309 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.885206 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.885218 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.885226 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.885280 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.891276 (+ 5996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.892866 (+ 1590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.892871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.892874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.893048 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.894301 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.894306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.894309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.894671 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.894895 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.895641 (+ 746us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.895968 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:59:10.897872 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.873763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.91.60:35269 (local address 127.0.0.1:54644)
0619 18:59:10.874145 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 18:59:10.874171 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:59:10.874316 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:59:10.875239 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:59:10.875246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:59:10.875884 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:59:10.876132 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.876145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.877487 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.877495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.879070 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:59:10.879089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.880603 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.880611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:59:10.880863 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.881625 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:59:10.881652 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:59:10.884910 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:59:10.891488 (+ 6578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.891495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.891499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.892698 (+ 1199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.893220 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:59:10.893228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:59:10.893234 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:59:10.893441 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:59:10.894860 (+ 1419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:59:10.894865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:59:10.895047 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 18:59:10.895250 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 18:59:10.895611 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260619 18:59:10.899937 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:10.827146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.15.91.62:46809)
0619 18:59:10.827602 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:10.827612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:10.831973 (+ 4361us) server_negotiation.cc:407] Connection header received
0619 18:59:10.832236 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:10.832245 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:10.832392 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:10.832511 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:10.851274 (+ 18763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.852514 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.858948 (+ 6434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:10.859717 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:10.859861 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:10.865384 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:59:10.865392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:59:10.865397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:59:10.865459 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:59:10.870346 (+ 4887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.871757 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.871765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.871771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.891866 (+ 20095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:59:10.893617 (+ 1751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:59:10.893624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:59:10.893627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:59:10.894129 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:59:10.897816 (+ 3687us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:10.899122 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0619 18:59:10.899448 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287}
I20260619 18:59:10.905236 6260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40: Bootstrap starting.
I20260619 18:59:10.910034 6260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40: Neither blocks nor log segments found. Creating new log.
I20260619 18:59:10.900345 6268 master_runner.cc:186] Error getting master registration for 127.15.91.61:35793: OK, instance_id { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" instance_seqno: 1781895550130197 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:59:10.911268 6268 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:59:10.911638 6260 log.cc:824] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40: Log is configured to *not* fsync() on all Append() calls
I20260619 18:59:10.918435 6379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:59:10.922515 6379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:59:10.923447 6260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40: No bootstrap required, opened a new log
I20260619 18:59:10.937254 6260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } }
I20260619 18:59:10.937777 6260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410b2dcffa8046918bae9b1277bf5b40, State: Initialized, Role: FOLLOWER
I20260619 18:59:10.939859 6260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } }
I20260619 18:59:10.945289 6382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } } }
I20260619 18:59:10.945895 6382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:10.946252 6310 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [term 1 FOLLOWER]: Refusing update from remote peer e9477a6d594443279482c09082c16af7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 18:59:10.946396 6289 tablet.cc:2369] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:59:10.947806 6344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:59:10.949000 6260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:59:10.949498 6234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:59:10.953843 6234 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [term 1 FOLLOWER]: Refusing update from remote peer e9477a6d594443279482c09082c16af7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:59:10.955188 6344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:59:10.957988 6235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9477a6d594443279482c09082c16af7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "410b2dcffa8046918bae9b1277bf5b40"
I20260619 18:59:10.964781 6365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9477a6d594443279482c09082c16af7. Latest consensus state: current_term: 1 leader_uuid: "e9477a6d594443279482c09082c16af7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } } }
I20260619 18:59:10.965243 6365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:10.982312 6366 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9477a6d594443279482c09082c16af7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } } }
I20260619 18:59:10.982663 6366 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:59:10.984519 6365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9477a6d594443279482c09082c16af7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } } }
I20260619 18:59:10.984908 6365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:10.985158 6379 catalog_manager.cc:1347] Generated new cluster ID: e881b3a556554e359cb7a1539937d1c0
I20260619 18:59:10.985271 6379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:59:10.996183 6382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9477a6d594443279482c09082c16af7. Latest consensus state: current_term: 1 leader_uuid: "e9477a6d594443279482c09082c16af7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } } }
I20260619 18:59:10.996603 6382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:59:11.000588 6235 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e9477a6d594443279482c09082c16af7 for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260619 18:59:11.003988 6344 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9477a6d594443279482c09082c16af7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } } }
I20260619 18:59:11.004554 6344 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:59:11.010999 6382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9477a6d594443279482c09082c16af7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9477a6d594443279482c09082c16af7" member_type: VOTER last_known_addr { host: "127.15.91.62" port: 46809 } } peers { permanent_uuid: "410b2dcffa8046918bae9b1277bf5b40" member_type: VOTER last_known_addr { host: "127.15.91.61" port: 35793 } } peers { permanent_uuid: "470638aac7b049b88e8b74bf1b24d83e" member_type: VOTER last_known_addr { host: "127.15.91.60" port: 35269 } } }
I20260619 18:59:11.011330 6382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 18:59:11.023981 6399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:59:11.024776 6399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:59:11.031298 6379 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:59:11.033537 6379 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:59:11.065855 6379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9477a6d594443279482c09082c16af7: Generated new TSK 0
I20260619 18:59:11.067148 6379 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:59:11.190019 6339 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.173236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.15.91.62:46809)
0619 18:59:11.173681 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.173688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.187671 (+ 13983us) server_negotiation.cc:407] Connection header received
0619 18:59:11.188052 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.188062 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.188156 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.188251 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.189657 (+ 1406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44066: BlockingRecv error: recv got EOF from 127.0.0.1:44066 (error 108)
Metrics: {"server-negotiator.queue_time_us":272}
W20260619 18:59:11.190025 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.172992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.15.91.61:35793)
0619 18:59:11.173227 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.173233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.187939 (+ 14706us) server_negotiation.cc:407] Connection header received
0619 18:59:11.188102 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.188107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.188204 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.188305 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.189709 (+ 1404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60894: BlockingRecv error: recv got EOF from 127.0.0.1:60894 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 18:59:11.191020 6338 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.179818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.15.91.60:35269)
0619 18:59:11.180026 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.180033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.188622 (+ 8589us) server_negotiation.cc:407] Connection header received
0619 18:59:11.188930 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.188936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.189027 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.189109 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.190661 (+ 1552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54654: BlockingRecv error: recv got EOF from 127.0.0.1:54654 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 18:59:11.200923 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.197386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.15.91.61:35793)
0619 18:59:11.198198 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.198205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.198919 (+ 714us) server_negotiation.cc:407] Connection header received
0619 18:59:11.199206 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.199213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.199377 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.199548 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.200586 (+ 1038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60908: BlockingRecv error: recv got EOF from 127.0.0.1:60908 (error 108)
Metrics: {"server-negotiator.queue_time_us":656}
W20260619 18:59:11.209069 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.206095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.15.91.61:35793)
0619 18:59:11.206301 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.206307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.207431 (+ 1124us) server_negotiation.cc:407] Connection header received
0619 18:59:11.207647 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.207653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.207759 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.207896 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.208801 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60918: BlockingRecv error: recv got EOF from 127.0.0.1:60918 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 18:59:11.223999 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.221252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.15.91.61:35793)
0619 18:59:11.221469 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.221475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.222174 (+ 699us) server_negotiation.cc:407] Connection header received
0619 18:59:11.222404 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.222411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.222538 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.222663 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.223691 (+ 1028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60920: BlockingRecv error: recv got EOF from 127.0.0.1:60920 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 18:59:11.240010 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.237296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.15.91.61:35793)
0619 18:59:11.237520 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.237527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.238240 (+ 713us) server_negotiation.cc:407] Connection header received
0619 18:59:11.238468 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.238475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.238599 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.238763 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.239735 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60924: BlockingRecv error: recv got EOF from 127.0.0.1:60924 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 18:59:11.255226 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.252548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.15.91.61:35793)
0619 18:59:11.252768 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.252775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.253448 (+ 673us) server_negotiation.cc:407] Connection header received
0619 18:59:11.253688 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.253695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.253800 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.253942 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.254956 (+ 1014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60940: BlockingRecv error: recv got EOF from 127.0.0.1:60940 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 18:59:11.278640 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.275635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.15.91.61:35793)
0619 18:59:11.275878 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.275887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.276694 (+ 807us) server_negotiation.cc:407] Connection header received
0619 18:59:11.276980 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.276986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.277099 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.277259 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.278343 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60944: BlockingRecv error: recv got EOF from 127.0.0.1:60944 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260619 18:59:11.318536 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.315909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.15.91.61:35793)
0619 18:59:11.316117 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.316125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.316765 (+ 640us) server_negotiation.cc:407] Connection header received
0619 18:59:11.316994 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.317000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.317098 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.317230 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.318251 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60952: BlockingRecv error: recv got EOF from 127.0.0.1:60952 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 18:59:11.398118 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.395355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.15.91.61:35793)
0619 18:59:11.395570 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.395576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.396155 (+ 579us) server_negotiation.cc:407] Connection header received
0619 18:59:11.396338 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.396344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.396456 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.396595 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.397841 (+ 1246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60954: BlockingRecv error: recv got EOF from 127.0.0.1:60954 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 18:59:11.532389 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.530074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.15.91.61:35793)
0619 18:59:11.530257 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.530262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.530804 (+ 542us) server_negotiation.cc:407] Connection header received
0619 18:59:11.531019 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.531024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.531104 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.531209 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.531969 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60958: BlockingRecv error: recv got EOF from 127.0.0.1:60958 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 18:59:11.795456 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:59:11.793406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.15.91.61:35793)
0619 18:59:11.793668 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:11.793675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:11.793925 (+ 250us) server_negotiation.cc:407] Connection header received
0619 18:59:11.794185 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:11.794190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:11.794269 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:11.794387 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:59:11.795267 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60968: BlockingRecv error: recv got EOF from 127.0.0.1:60968 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 18:59:11.891023 6377 catalog_manager.cc:1259] Loaded cluster ID: e881b3a556554e359cb7a1539937d1c0
I20260619 18:59:11.891145 6377 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e: loading cluster ID for follower catalog manager: success
I20260619 18:59:11.894419 6377 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e: acquiring CA information for follower catalog manager: success
I20260619 18:59:11.896941 6377 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 470638aac7b049b88e8b74bf1b24d83e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:59:12.026814 6399 catalog_manager.cc:1259] Loaded cluster ID: e881b3a556554e359cb7a1539937d1c0
I20260619 18:59:12.026942 6399 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40: loading cluster ID for follower catalog manager: success
I20260619 18:59:12.029768 6399 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40: acquiring CA information for follower catalog manager: success
I20260619 18:59:12.031988 6399 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 410b2dcffa8046918bae9b1277bf5b40: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:59:12.318969 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:12.312284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.15.91.61:35793)
0619 18:59:12.313103 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:12.313108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:12.313127 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:59:12.313197 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:12.313201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:12.313380 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:12.313501 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:12.313984 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:12.314851 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:12.316343 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:12.316768 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:12.316883 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:12.317572 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:12.318156 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:12.318282 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:12.318465 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 18:59:12.318549 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":589,"threads_started":1}
I20260619 18:59:12.332368 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:12.325740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.15.91.60:35269)
0619 18:59:12.326324 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:12.326328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:12.326350 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:59:12.326572 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:12.326576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:12.326824 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:12.327011 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:12.327434 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:12.328247 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:12.329831 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:12.330294 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:12.330408 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:12.331085 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:12.331674 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:12.331819 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:12.331975 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 18:59:12.332059 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":293,"threads_started":1}
I20260619 18:59:12.349411 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:12.339232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.15.91.60:35269)
0619 18:59:12.339444 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:12.339451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:12.342781 (+ 3330us) server_negotiation.cc:407] Connection header received
0619 18:59:12.343054 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:12.343060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:12.343164 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:12.343277 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:12.344008 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:12.345186 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:12.347097 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:12.347713 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:12.347885 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:12.348410 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:12.348557 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:12.348648 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:12.348889 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 18:59:12.349001 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:59:12.349602 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:12.339225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.15.91.61:35793)
0619 18:59:12.339436 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:12.339445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:12.341997 (+ 2552us) server_negotiation.cc:407] Connection header received
0619 18:59:12.342458 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:12.342464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:12.342570 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:12.342640 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:12.343733 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:12.344949 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:12.347433 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:12.348090 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:12.348223 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:12.348725 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:12.348880 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:12.348976 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:12.349184 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 18:59:12.349292 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:59:12.352154 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:59:12.339238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.15.91.62:46809)
0619 18:59:12.339881 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0619 18:59:12.339892 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:59:12.342340 (+ 2448us) server_negotiation.cc:407] Connection header received
0619 18:59:12.342492 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:59:12.342497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:59:12.342790 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:59:12.342997 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:59:12.343960 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:12.344797 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:12.348384 (+ 3587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:59:12.349051 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:59:12.349188 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:59:12.350759 (+ 1571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:59:12.351278 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:59:12.351431 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 18:59:12.351826 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 18:59:12.351911 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":370,"threads_started":1}
I20260619 18:59:12.358541 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 6119
I20260619 18:59:12.359977 6181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:59:12.474262 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 6119
I20260619 18:59:12.497506 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 6193
I20260619 18:59:12.498662 6255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:59:12.609195 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 6193
I20260619 18:59:12.631045 15724 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskualRwQ/build/asan/bin/kudu pid 6268
I20260619 18:59:12.632325 6330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:59:12.744930 15724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskualRwQ/build/asan/bin/kudu with pid 6268
2026-06-19T18:59:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5043 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10247 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (388154 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST